Manuel Vidaurre-Arenas, Ramón Martínez-Piedra, PAHO, Health Analysis and Information Systems (DD/AIS); Carlos Castillo-Salgado, PAHO, Special Advisor, Forum for Public Health in the Americas (D/DPM).
It is not because countries are poor that they cannot afford good health information.
It is because they are poor that they cannot afford to be without it.
Health Metrics Network
What is the Health Metrics Network?
Health Metrics Network (HMN) is a global partnership that will work to increase the availability and use of timely and trustworthy health information in developing countries and globally. This partnership supports the use of a common framework and guidelines for assessing, strengthening and monitoring health information systems, for better decision-making and contributes to the development of new knowledge related to health information systems (HIS) development and management. HMN is instituted on the premise that better health information means better decision making and therefore better health for all. HMN partners are working to improve health and save lives by strengthening and aligning investments in the development of HIS in accordance with the broader global development agenda, including the Millennium Development Goals, around the world. HMN has also a direct key investment mechanism through grants targeting low and middle income countries where major needs for improving health information and decision-making, strongest government commitment for improving and changing current process and practices, and the most supportive partners are in place. This global collaboration is designed to deliver long-lasting solutions to HIS development. The website of HMN is available in http://www.who.int/healthmetrics.
Who is the HMN?
HMN was launched in the 58th World Health Assembly in 17 May 2005.1 The HMN Board is constituted by the African Population and Health Research Center; Bill & Melinda Gates Foundation; Centers for Disease Control and Prevention of the U.S. Department of Health and Human Services; Danish International Development Agency; Department for International Development of United Kingdom (DFID); European Commission; Ghana Health Services; Global Fund to fight AIDS, Tuberculosis and Malaria; Ministry of Health, Mexico; Ministry of Public Health, Thailand; Organization for Economic Co-Operation and Development; Statistics South Africa; Uganda Bureau of Statistics; UNICEF; United Nations Statistics Division; U.S. Agency for International Development (USAID); World Bank; and World Health Organization.2,3
The statement of support for HMN4 was signed by the HMN Board of another four agencies: Global Alliance for Vaccines and Immunization (GAVI); Royal Danish Ministry for Foreign Affairs;5 Swedish International Development Agency; United Nations Population Fund.
HMN has an organizational structure composed of a Board, a Technical Advisory Group (TAG), an Independent Review Committee (IRC), and a small Secretariat hosted by WHO. More about this structure is described in following sections.
How the HMN will operate?
HMN received a grant of US$ 50 million over seven years from the Bill & Melinda Gates Foundation6,7 and additional contributions from other donors including the Department for International Development of the United Kingdom,8 U.S. Agency for International Development,9 Danish International Development Agency, Goverment of Thailand and other multilateral and bilateral agencies.
HMN will meet its objectives through a variety of actions and activities. The main target for investments is low- and middle- income countries, which may be eligible to apply for grants of up to US$ 500,000 for the strengthening of HIS and can call upon HMN partners for technical assistance using the HMN framework.
When will the HMN operate?
From now until 2011, HMN expects to document improved health outcomes that can be realistically attributed to the enhanced use of information for decision making in at least thirty developing countries. Additional funding will be available to further support this result based global partnership and align investments in HIS in harmony with the global development agenda. The requests for country applications (RFCA)10,11 will be made anually. Detailed information about this process can be found in:
http://www.who.int/healthmetrics/about/en/.
Where will the HMN operate?
Low and middle income countries with the greatest need in terms of health information, the strongest commitment to improvement and change, and the most supportive partners will be the priority for substantive financial support. According to HMN documents, in general, countries will fall into one of three categories:10
1. The group of countries of highest priority for HMN action is low and lower-middle income countries. These countries would be eligible for considerable, sustained, long-term financial and technical support.They may apply for HMN grants of up to US$ 500,000 in a given year for HIS strengthening including consensus-building, strategy development, planning, resource mobilization, assessment and monitoring. Countries in this group may be able to access HMN technical and financial support over several years even though it is anticipated that the main volume of HMN financial support would occur in the diagnostic, developmental and planning phases with in-country donors generating the bulk of the resources for implementation.
2. Middle and upper middle income countries will be eligible for technical assistance plus limited financial support of up to US$100,000 in a given year to enable convening of partners, focused strengthening of specific elements of the health information system, experience sharing, or time-limited operations research. Countries in this group are likely to focus on specific aspects of the health information system identified as in need of strengthening.
3. All countries except high income countries are eligible for focused technical assistance in specific health information system areas for which they are able to generate their own in-country financial funds with no expectation of additional resources from HMN. These countries are likely also to serve as regional centers of excellence and inter-country technical support and to facilitate sharing of best practices and learned lessons.
Countries are eligible to apply for an extension and/or expansion of existing HMN support on the basis of a review and evaluation of progress accomplished on ongoing needs. Some countries will receive support over a period of several years, five to ten, to shape national Health Information Systems plans and policies.
What is the Goal and Objectives of HMN?
HMN is set up as an interactive partnership between countries and global actors, not a top-down strategy, and its main focus is building country systems to ensure long-term sustainability of sound data generation and use. Its goal and objectives are:
Goal:12 To increase the availability and use of timely and reliable health information in countries and globally through a shared agreement on the goal and coordinated investments in core health information systems.
Objectives:
1. Develop a framework and standards for health information systems (Consensus Guidelines).
2. Support countries in applying the HMN framework (Country Implementation).
3. Develop incentives for enhanced dissemination and use of sound health information (Universal Access).
To increase the availability and use of timely and accurate health information by catalyzing the joint funding and development of core health information systems is the overarching strategic goal of HMN. To accomplish this goal, HMN is laying out a vision; identifying strategies for health information systems development and strengthening; supporting countries in strategies implementation; and generating new knowledge and global public good through research, technical innovation, and sharing lessons learned.
As mentioned above, HMN will pursue three interrelated objectives. First, create a consensus framework for the country health information system assessment, development and monitoring that describes standards for health information systems; second, strengthen country health information systems by offering technical and catalytic financial support for the application of the HMN framework; and third, improve accessibility and use of health information through policies, systems and incentives to ensure the access and utilization of information at local, regional and global levels.
What are the Guiding Principles, Values and Practices of the HMN?
The HMN main principles are country leadership and linking health information system development with wider efforts to improve and evolve national statistical capacities according to agreed standards, such as the Fundamental Principles of Official Statistics adopted by The United Nations Statistical Commission, in its Special Session of 11-15 April 1994.13 Building coherent health information systems requires a long-term investment and should be seen as a gradual, incremental process. This will help ensure long-term sustainability and the buy-in of multiple partners including those in non-health sectors.
The process of health information system development is ongoing. As country partners work together to strengthen the health information system, they will draw upon the growing body of knowledge about health information systems development generated through HMN activities. In turn, experiences across many countries will contribute to the further elaboration of the HMN framework at the global level and its adaptation and use in other countries.
What are the governance mechanisms in HMN?
HMN has an organizational structure (figure 1) composed of a Board, a Technical Advisory Group (TAG), an Independent Review Committee (IRC), and a small Secretariat.14 The members of the Board are representatives from HMN’s constituencies, selected on the basis of commitment during the development phase of HMN. New Board membership is reviewed by the Board based on evolving needs. The Board provides leadership and strategic guidance to HMN, approves the overall budget and work plan prepared by the Secretariat, and approves the annual report presented by the Secretariat. The Board appoints the members of the TAG, IRC, and other committees it may establish.
The TAG provides guidance on the further development of the harmonized framework and standards for country health information system development, its assessment and monitoring tool; develops guidelines for country solicitations of support from HMN; advises on country support including governance and coordination mechanisms and monitoring of country progress; among other technical advisory responsibilities.
The IRC advises the Board on the selection of countries for HMN support. The recommendations of the IRC are transmitted to the HMN Board through the Secretariat. Country proposals are submitted to the IRC through the Secretariat which ensures that applications meet the administrative requirements outlined in the application guidelines. Technical feedback from the IRC on country proposals may be transmitted to country teams through the Secretariat.
What is the HMN Framework?
The HMN Framework (HMNF) serves to focus partner actions and guide the overall direction of HIS reform. The HMNF conveys varied data generation and gathering approaches within a unifying framework and therefore helps to define country and global systems, standards, capacities and processes needed for health information system strengthening. This is needed because one of the major constraints to strengthening health information systems is the absence of consensus on the relative strengths, usefulness and feasibility of different data collection approaches. The HMNF tries to bring together a normative framework for measurement in health with participatory assessment, planning and implementation modalities that should be objective, transparent and include all stakeholders.
It is expected that the HMNF will serve several purposes: as a diagnostic tool, as a roadmap, as an accreditation tool, and as a tool to focus investment and technical assistance for HIS
The HMN framework15 has two parts: a normative component and an implementation component (figure 2).
1. The normative component describes the standards and assessment criteria linked to the inputs, processes, outputs and outcomes of the health information system and encompasses six sub-components:
a) Context and resources for health information. The policy, legislative, regulatory, and financial environment that must be in place; and the infrastructure and resources required to guarantee a wholly functional health information system.
b) Identification of core health indicators, defining core health indicators covering the domains of health information, grouped in four main types: determinants of health, health system inputs and outputs, health system outcomes, and health status (figure 3).
c) Data sources and data collection methods, key data sources, standards for their use, their role in generating health information and potential linkages between them. The subsystems are census, vital events monitoring, health facilities statistics, public health surveillance, population-based surveys and resource tracking, including health infrastructure and human resources (figure 4).
d) Information management processes, optimal processes for collecting, sharing, and storing data, data flows and feedback loops.
e) Data availability and quality, criteria for assessing the quality of available data.
f) Data dissemination and use, norms for presenting, disseminating data and sharing information among stakeholders and creation of incentives for evidence-based decision making.
2. The implementation component outlines a roadmap for strengthening health information systems and includes a tool to guide situation analysis of the country HIS, thus facilitating countries to establish a baseline and monitor progress of HIS development. This process is linked to the set of principles, processes and benchmarks for implementation of the HMN framework at the country level. The HMN principles include country leadership and ownership, consensus-building, focus on country needs, and HIS development as a gradual and incremental process. The process will clearly define stages and benchmarks and will state the specific role of HMN as a catalyst and technical resource.
As a recommendation of the HMN Board for refining, testing and improving the HMNF and tools, three pathfinder countries have been identified for the first phase: Ghana, Mexico, and Thailand. And the HMN Board mandated the Secretariat to identify two additional pathfinder countries.17 Pathfinder countries are identified on the basis of expressed desire to support the development phase of HMN and capacity and interest to be involved in the initial development and testing. The identification of a country as pathfinder will not preclude it from submitting a proposal for HMN support through the normal country submissions process.18
What are the components of a proposal for HMN?
For the requests for country applications (RFCA) in 2005 a proposal10 for HMN consisted of:
a) Formal statement of common purpose by key country stakeholders including high-level political commitment to the HMN principles and a willingness to engage in long-term commitment;
b) Summary description of major strengths and weaknesses of current country health information system;
c) Summary of existing national plans for HIS, if any, and/or statistical development;
d) Outline of type of assistance sought (technical support, financial support for development of HIS plan, catalytic support to HIS development);
e) Statement of key deliverables and timeline;
f) Composition of country teams, specifically detailing the involvement and commitment of key partners at senior levels including Ministry of Health, disease-specific program managers, National Statistics Office, country institutes of public health or equivalent, research institutes, private health care providers, donors, multilateral agencies, and technical experts; and
g) Potential resources available in-country and from donors.
It is anticipated that future rounds will use a slightly modified version of the proposal guidelines based on experiences during the first round.
Additional information may be found in: http://www.who.int/healthmetrics/about/applications/en/index.html
How countries can apply for support from HMN?
Applications for HMN support will be assessed through an open and transparent process with all proposals submitted for review by the Independent Review Committee (IRC)10,11 which will recommend countries for support on the basis of the following criteria:
- Countries in greatest need (measured by level of under-five mortality and GDP per capita) where HIS development is likely to have the greatest potential to enhance the efficiency and effectiveness of health development efforts;
- Completeness and overall coherence of the proposal;
- Involvement and commitment of country partners, in particular, links between Ministry of Health and National Statistics Offices, as well as donors and development agencies;
- Form and functions of country mechanisms for coordination and disbursement of funds; and
- Anticipated results and impact, including sustainability.
Background information on the country health information system, in particular with regard to availability and quality of reporting on key indicators, will be provided by the Secretariat to support the IRC in making its review and assessment.
What is the PAHO/AIS position regarding HMN?
PAHO, as the WHO Regional Office for the Americas, supports the goal, values and principles of HMN, but incorporates the regional perspective, with emphasis on strengthening and expanding already successful current initiatives used such as inputs, as the Core Health Data Initiative18,19,20 that had been supported by PAHO since 1994, and key national HIS experiences already established in the Americas.
An important issue after an HIS assessment is the selection of sound initiatives and strategies to overcome weaknesses identified as priorities. In this regard, PAHO in partnership with MEASURE/EvaluationB and USAID/LAC, together with national teams is carrying out a project for documenting lessons learned, successful stories and key processes in HIS including two case studies in the Region: Brazil and Mexico. The overarching goal of this project is to develop a working standardized framework and evidence for supporting the selection of appropriate strategies and solutions in context for overcoming specific critical elements in HIS. This working framework has been developed based on the documentation of the experience of Brazil and Mexico in HIS. In these countries, under a country-driven process based on the HMN framework and instruments, a more comprehensive in-depth assessment has been carried out using additional methods and instruments. To complement the situation analysis and monitoring tool (SAMT, formerly HAM), two additional tools have been specifically selected for implementation: the Organizational and Behavioral Assessment Tool (OBAT)21 part of the MEASURE/Evaluation Performance Routine Information Systems Management Framework (PRISM)22 tool package; and the Assessment for Health Enterprise Architecture Development (AHEAD,23 formerly EIA2Tool) from PAHO/AIS.
The OBAT intends to identify strengths and weaknesses of HIS performance, including the analysis of need components (correlates) to produce good quality of data and sustained uses of health data for improving health system operations and population health status. In the PRISM,22 the determinants for Routine Health Information Systems (RHIS) performance are classified in three groups:
a) technical determinants: data quality, system design, IT, etc.;
b) organizational/environmental determinants: information culture, structure, roles and responsibilities, resources; and
c) behavioral determinants: knowledge/skills, attitudes, values, motivation, etc. (figure 5)
Traditionally, for assessing RHIS only technical determinants are taken in consideration. With the addition of behavioral and organizational determinants of RHIS, the tool draws attention to the important role played by the presence/absence of a culture of information and by the existing knowledge, practices and motivation of staff. Its two main objectives are:
a) to assess the role of behavioral and organizational factors on HIS performance; and
b) to develop strategies based on gaps and weaknesses identified in behavioral and organizational determinants through the assessment.
The AHEAD framework aims to integrate a maturity model,C,24,25 for HIS to enable an adaptable Enterprise Architecture (EA)D,26,27,28,29,30,31 to facilitate the interoperability or integration of current/new HIS, ensuring its appropriate performance aligned to overarching goals of Ministries of Health and key National Health Institutions (social security). One of the key issues of current HIS is its isolation and limited possibilities to be easily integrated or interoperable with other HIS in the Ministries of Health.32 A silo culture with compartmentalized vertical HIS by programs precludes the possibility of real use of health data and information to support decision-making in important public health programs. To respond better to these challenges the strengthening and development of HIS using an integrated solution as EA is being used in Canada with the initiative InfowayE. An EA is a formal description of services and technology options based on standards and guidelines for systems and subsystems organized by structural and functional properties to fulfill HIS goals. The EA describes the HIS or building blocks, defining their interrelationships in a detailed plan, from which a HIS can rapidly incorporate new functionality, being adaptable to changing environments.
AHEAD includes an ordered set of maturity levels with reference elements that describe effective processes, practices and protocols that generate a road map. This can provide guidance for future investments, supporting the resolution of gaps in the organization’s business and information functions. AHEAD contains a rapid assessment and monitoring tool based on a Generalized Enterprise Architecture Model for assessing the current maturity level of the EA. The maturity model can be used as a benchmark for assessing different EA, organizations or HIS for equivalent comparison. The maturity model follows the path of a Ministry of Health as their health EA program matures, and sets benchmarks to measure performance towards progression in the development of its health EA. The AHEAD Generalized Enterprise Architecture Model covers five layers: infrastructural, architectural, organizational, informational, and business/domain.
The AHEAD assessment tool is based on:
* ISO/IEC 1550433,34 emerging international standard for software process assessment;
* Carnegie Mellon, Software Engineering Institute, Standard Capability Maturity Model Integration® Appraisal Method for Process Improvement (SCAMPISM);35
* National Association of State Chief Information Officers (NASCIO). Enterprise Architecture Development Tool-Kit36
* US Office of Management and Budget, Enterprise Architecture Assessment Guidelines37,38
The OBAT and the AHEAD tools will complement the findings from the application of HMNF-SAMT. In the next two years, results from the above case studies and further evolution of the framework and tools will be available for use of PAHO Member States. This novel experience will help them to plan and prioritize investments to strengthen HIS and its use for better decisions in public health.
PAHO maintains a close collaboration with the HMN Secretariat and several TAG members to obtain and provide feedback in the application of this framework and tools. Within the process of strengthening HIS, important activities are under way. In November, the first Regional Meeting of National Directors of Statistics and National Directors of Health Statistics will take place in Buenos Aires, Argentina, to identify priority areas for collaboration. As part of the Regional HMN process, the organization of a workshop for writing successful proposals for HMN, for priority countries in the Region, is in the planning phase.
The PAHO’s Transformation Roadmap for Organizational Change incorporates the initiative of the Forum of Public Health in the Americas (FPHA) to create a regional platform for internal and external constituents to dialogue, debate, and learn about important health topics. The FPHA will include key topics for the debate and dialogue of health metrics as an essential basis to support policy and prospects for Public Health in the XXI century.
More information regarding the above will be provided in coming issues of the PAHO Epidemiological Bulletin.
Acknowledgments:
The authors would like to thank Gabriela Fernandez, Enrique Loyola and Patricia Najera for the revision of this paper. We acknowledge the work of our colleagues in WHO/HQ, regarding Health Metrics Network, Carla Abou-Zahr for her significant contributions to this paper and Ties Boerma, as well their technical advisors. We recognize the hard work of national teams in Brazil and Mexico, under the leadership of Pedro Benevenuto Jr and Rafael Lozano respectively. As well, we acknowledge our partners in MEASURE/Evaluation, Beatriz Plaza and Theo Lippeveld, and Kelly Saldana from USAID/LAC.
A Several portions of this paper were adapted from the HMN Concept Paper: What it is, what it will do and how countries can benefit.
http://www.who.int/healthmetrics/about/concept_paper.doc
B http://www.cpc.unc.edu/measure/
C A maturity model is a structured collection of elements that describe characteristics of effective processes, practices, disciplines and protocols, which is ordered as levels with specific sets of elements that an organization must fulfill to earn a specific maturity level.
D Enterprise Architecture is a comprehensive framework used to manage and align an organization’s structure, processes, information, operations and projects with the organization’s overall strategy. Ultimately, this structured process helps to guide an organization to make sound, targeted decisions about how to manage and structure its information-related assets for maximum effectiveness. http://en.wikipedia.org/wiki/Enterprise_architecture accessed on 27 September 2005
E http://knowledge.infoway-inforoute.ca/ehr_blueprint/en.asp
References:
1. LEE Jong-wook. WHO Director’s speech: Health Metrics Network Launch. Geneva, Switzerland: WHO/HQ; 17 May 2005.
http://www.who.int/dg/lee/speeches/2005/healthmetricsnetwork/en/ accessed on 15 September 2005.
2. McNab, C. WHO Press Release: New global partnership will focus on strengthening health information systems to better address health needs worldwide. Geneva, Switzerland: WHO/HQ; 17 May 2005. http://www.who.int/mediacentre/news/releases/2005/pr20/en/ accessed on 15 September 2005.
3. Epstein, D. PAHO Press Release: World Health Assembly Hears Warning on Avian Influenza Threat. Washington DC: PAHO; 17 May 2005.
http://www1.paho.org/English/DD/PIN/pr050517.htm accessed on 15 September 2005.
4. Health Metrics Network. Statement of Support: Health Metrics Network. Geneva, Switzerland: WHO/HQ; 17 May 2005.
http://www.who.int/healthmetrics/about/statement_support_en.pdf accessed on 15 September 2005.
5. Iversen, HR. Health Metrics Network Launch: Statement by HE Mr. Henrik Rée Iversen, Permanent Representative of Denmark, Permanent Mission of Denmark, Geneva, Switzerland: WHO/HQ; 17 May 2005. http://www.missionfngeneve.um.dk/en/menu/STATEMENTS/HealthMetricsNetworkLaunch.htm accessed on 15 September 2005.
6. Bill & Melinda Gates Foundation. Global Health, Advocacy Grant: World Health Organization.
http://www.gatesfoundation.org/globalhealth/otherinitiatives/advocacy/grants/grant-31044_01.htm. accessed on 15 September 2005.
7. Gates, B. 2005 World Health Assembly: remarks addressed by Bill Gates to WHA. Geneva, Switzerland. WHO/HQ; 16 May 2005.
http://www.gatesfoundation.org/MediaCenter/Speeches/BillgSpeeches/BGSpeechWHA-050516.htm accessed on 15 September 2005.
8. Department for International Development (DFID). Contracts Let by Procurement Department April 2005. http://www.dfid.gov.uk/procurement/contracts2005april.asp accessed on 15 September 2005.
9. United States Agency for International Development (USAID). Global Health: Implementing Agencies and Global Partners.
http://www.usaid.gov/our_work/global_health/hs/partnerships/index.html accessed on 15 September 2005
10. Health Metrics Network. Guidelines for Country Applications. Geneva, Switzerland: WHO/HQ. 2005 http://www.who.int/healthmetrics/about/country_proposals.doc accessed on 15 September 2005
11. Health Metrics Network. Provisional Country Application for HMN Support. Geneva, Switzerland: WHO/HQ. 2005.
http://www.who.int/healthmetrics/about/hmn_proposal_guidelines_28april05.doc accessed on 15 September 2005
12. Health Metrics Network. [slide show], Health Metrics Network. Geneva, Switzerland. WHO/HQ. 29 April 2005.
http://www.who.int/entity/healthmetrics/about/hmn_29april05.ppt accessed on 15 September 2005
13. United Nations, Statistical Commission. Fundamental Principles of Official Statistics. UN; April 1994 http://unstats.un.org/unsd/methods/statorg/FP-English.htm accessed on 15 September 2005
14. Health Metrics Network. Health Metrics Network (HMN) Governance. Geneva, Switzerland: WHO/HQ; 15 April 2005.
http://www.who.int/healthmetrics/governance/governance_functions_final.pdf accessed on 15 September 2005
15. Health Metrics Network. Strengthening Country Health Information Systems: Situation Analysis Tool [Draft 1]. Geneva, Switzerland: WHO/HQ; 18 August 2005
16. Health Metrics Network. Summary of Decisions: 3rd Board meeting. Geneva, Switzerland. Palais des Nations ; 18 May 2005.
http://www.who.int/healthmetrics/governance/summary_decisions_31may.doc accessed on 15 September 2005
17. Health Metrics Network. Summary of Decisions: Board meeting 27-28 January 2005. Geneva, Switzerland: WHO/HQ; 21 February 2005.
http://www.who.int/healthmetrics/governance/summary_decisions_21feb.doc accessed on 15 September 2005
18. Pan American Health Organization. Collection and Utilization of Core Data in Health. Washington, DC: PAHO; 14 July 1997 (CD40/19 document http://www1.paho.org/english/gov/cd/doc165.pdf and CD40/R10 Resolution http://www1.paho.org/English/GOV/CD/ftcd_40.htm#R10). accessed on 15 September 2005.
19. Pan American Health Organization. Health Indicators: Basic Elements for the Health Situation Analysis. Epidemiological Bulletin PAHO, 2001; 22 (4): 1-5. http://www1.paho.org/english/dd/ais/eb_v22n4.pdf accessed on 15 September 2005.
20. Pan American Health Organization. 10-Year Evaluation of the Regional Core Data Initiative in Health. Washington, D.C.: PAHO; 1 October 2004. (Document CD45/14) http://www1.paho.org/english/gov/cd/CD45-14-e.pdf accessed on 15 September 2005.
21. Aqil A, Hozumi D, Lippeveld T, Organizational and Behavioral Assessment Tool [Draft]. MEASURE/JSI: 28 January 2005.
22. LaFond A, Fields R, Lippeveld T. The Prism: Introducing an Analytical Framework for Understanding Performance of Routine Health Information Systems in Developing Countries. In: Routine Health Information Network, Second International RHINO Workshop on: Enhancing the Quality and Use of Routine Health Information at District Level. Mpekweni Sun Eastern Cape, South Africa: RHINO, MEASURE/Evaluation, USAID, JSI and The Equity Project, 2003: 20-26. http://www.cpc.unc.edu/measure/publications/pdf/ws-04-12.pdf accessed on 15 September 2005.
23. Vidaurre M, Martinez R, Castillo-Salgado C. Assessment for Health Enterprise Architecture Development [Draft]. Washington, DC: PAHO; 14 October 2005.
24. Shihong Huang , Scott Tilley, Towards a documentation maturity model, Proceedings of the 21st annual international conference on Documentation, October 12-15, 2003, San Francisco, CA, USA
25. Pfleeger, S.L. , (1995), Maturity, Models, and Goals: How to Build a Metrics Plan, Journal of Systems and Software, 31, 143-155 (1995).
26. Janssen M, Cresswell A. Enterprise Architecture Integration in E-government. Proceedings of the 38th Hawaii International Conference on System Sciences – 2005. Hawaii: IEEE; 2005. http://csdl2.computer.org/comp/proceedings/hicss/2005/2268/05/22680118b.pdf [requires subscription] accessed on 15 September 2005.
27. Armour FJ, Kaisler SH. Enterprise Architecture: Agile Transition and Implementation, IT Professional, 2001; 6(3): 30-37.
http://csdl.computer.org/dl/mags/it/2001/06/f6030.pdf [requires subscription] accessed on 15 September 2005.
28. Armour FJ, Kaisler SH, Liu SY. A Big-Picture Look at Enterprise Architectures. IT Professional, 1999; 1(1): 35-42.
http://csdl.computer.org/dl/mags/it/1999/01/f1035.pdf accessed on 15 September 2005.
29. Armour FJ, Kaisler SH, Liu SY. Building an Enterprise Architecture Step by Step. IT Professional, 1999; 1(4): 31-39.
http://csdl.computer.org/dl/mags/it/1999/04/f4031.pdf accessed on 15 September 2005.
30. Zachman JA., A framework for information systems architecture, IBM Systems Journal, v.26 n.3, p.276-292, 1987.
http://www.research.ibm.com/journal/sj/263/ibmsj2603E.pdf accessed on 15 September 2005.
31. National Association of State Chief Information Officers (NASCIO). NASCIO Enterprise Architecture Maturity Model, Version 1.3. US: NASCIO; December 2003. https://www.nascio.org/hotIssues/EA/EAMM.pdf accessed on 15 September 2005.
32. Coordinación General de Planeación Estratégica. Dirección General de Información y Evaluación del Desempeño. Programa de Acción: Sistema Nacional de Información en Salud 2001-2006. México, D.F.: Secretaría de Salud; 2003. http://www.salud.gob.mx/docprog/estrategia_7/sinais.pdf accessed on 15 September 2005.
33. Carnegie Mellon University, Software Engineering Institute (SEI). An International Collaboration to Develop a Standard on Software Process Assessment. http://www.sei.cmu.edu/iso-15504/ accessed on 15 September 2005.
34. International Organization for Standardization. SPICE Network. ISO/IEC 15504:2004 Information technology -- Process assessment -- Part 1: Concepts and vocabulary. http://isospice.com/standard/is15504.htm accessed on 15 September 2005.
35. SEI Joint Program Office, CMM IntegrationSM Project. Standard CMMISM Appraisal Method for Process Improvement (SCAMPISM), Version1.1: Method Definition Document. Hanscom AFB, MA: SEI; 2001. [CMU/SEI-2001-HB-001] http://www.sei.cmu.edu/pub/documents/01.reports/pdf/01hb001.pdf accessed on 15 September 2005.
36. National Association of State Chief Information Officers (NASCIO). Enterprise Architecture Development Tool-Kit. US: NASCIO, version 3. Kentucky: NASCIO; October 2004. https://www.nascio.org/nascioCommittees/ea/toolkitDownload.cfm accessed on 15 September 2005.
37. OMB FEA Program Management Office. Guidelines for Enterprise Architecture Assessment Framework. US: OMB; April 2004.
http://www.feapmo.gov/resources/040427%20EA%20Assessment%20Framework.pdf accessed on 15 September 2005.
38. OMB FEA Program Management Office. OMB Enterprise Architecture Assessment v1.0 Guidelines. US: OMB; April 2004.
http://www.feapmo.gov/resources/OMB%20Enterprise%20Architecture%20Assessment%20v1.0.pdf accessed on 15 September 2005.