Within the last decade many projects have been implemented in order to provide Technical Assistance (TA) to countries, which have initiated their transition from centrally planned to market economies (such as some countries in Eastern Europe). The pace and the degree of success of this transition, which these economies and - most important - their societies are undergoing, will certainly have an impact on the global economy, politics, social security, democracy and peace.

Monitoring and evaluation (M&E) exercises assist these TA projects in achieving their objectives, by providing management with information on the projects' implementation, so that structured decisions can be taken, if and when needed. In this framework, it became obvious that an overall M&E database system was needed.

An Integrated Monitoring & Evaluation System (IMES) was developed for the needs of the European Commission (DG1A).Its main objective was to improve management reporting, by extracting relevant information on the performance of projects from various monitoring and evaluation reports and by producing overall statistics at NIS (New Independent States) level.

Contents

The Role of Technical Assistance Programmes & Projects
Monitoring & Evaluating Technical Assistance
IMES
Operational Capabilities of IMES
Evolution of IMES
Application of IMES
Current Use of IMES
Conclusions

The Role of Technical Assistance Programmes & Projects

Within the last decade many projects have been implemented in order to provide Technical Assistance (TA) to countries, which have initiated their transition from centrally planned to market economies (such as countries in Eastern Europe previously under the Soviet sphere of influence). The basic role of these programmes (such as Tacis, Phare and USAID) is to assist and support these countries in achieving their transition targets and to strengthen their democratic societies, based on political freedom and economic prosperity. Technical Assistance Programmes, consisting of a comprehensive series of projects created to address the countries' needs, provide support in the form of grants to foster an exchange of knowledge and expertise through partnerships, links and networks at all levels of society. This know-how is delivered in a number of ways, known as the Programmes' Techniques. These are:

To provide:	To develop:	To set up:
policy advice advisory & consultancy teams training studies	new legal and regulatory frameworks new & reformed institutions non-governmental organisations	partnerships and networks twinning pilot projects framework programmes for fast interventions in specific key areas

Monitoring & Evaluating Technical Assistance

The "Monitoring & Evaluating" function (M&E) is a project management tool that helps check each TA project to see if it is being implemented correctly. M&E permits informed project management decisions to be made, allowing adjustments to occur with minimal disruption. It essentially keeps projects on course so they can reach their objectives.

M&E teams, which carry out systematic on-the-ground monitoring and evaluation, aim at improving project performance by providing, in a timely fashion, relevant information and recommendations to the Programmes' services.

The most important monitoring and evaluation activities during the various phases of each TA project are:

Project Launching Phase	Start-up and Implementation Phase	Completion Phase
Receiving / obtaining information on : ToR, SoE, Objectives, Wider results, impact expected	Attendance to project meetings (kick-off, Inception, Progress) and events (seminars, presentations, etc.)	Attendance to the End-of-Project presentation
Benchmark dates Contractor selected	Studying project's Technical & Management Reporting	Studying the Final Report and Project Completion Report
Organisation and methods applied Programming of activities & outputs	Implementation of field visits : meetings & interviews with the key actors (contractor, Project Partner, other) & target groups	Implementation of the End-of-Project Assessment field visit : interviews with the key actors and target groups
Preliminary contacts with the Project Task Manager	Assessment of the Project Progress, as compared to the specifications and planning set	Assessment of Project Performance, Appropriateness for the Partner, Impact, Replicability, Dissemination of results
Initial planning of the monitoring activities	Preparation of monitoring reports with standardised content and format and dissemination of the reports to all specified recipients of theirs, within clearly set time limits
	Follow-up of the project's progress between monitoring visits	Implementation of the Evaluation visit, at a specified time after project completion
		Evaluation of the project's impact and of the sustainability of its results and preparation / dissemination of an Evaluation report (as above)

IMES

General

IMES is a dynamic monitoring & evaluation system, developed for the needs of the European Commission (DG1A). Its operation is focused on the M&E exercise implemented in the NIS (New Independent States). Its main objectives are to improve the progress and results of Tacis and to provide support for the management of Tacis, by allowing the:

• extraction of relevant information on projects' performance from the monitoring and evaluation reports; and,
• production of overall statistics at NIS level

Management support is mainly concerned with future TA projects as well as the release and allocation of funds; the maximization (or minimization) of technical assistance to specific regions, countries or sectors; and, measures needed to resolve specific problems in certain areas.

IMES Architecture

IMES is an integrated information system which incorporates Internet technologies in order to provide broad monitoring and evaluation capabilities. It consists of five individual, but interacting, subsystems which form a robust Intranet information system. The structure of the system is presented in the Figure 1.

Figure 1: Structure of IMES

The Information System Database

The Database is the "back end" application that is used for storing all kinds of data. It is built with Microsoft (MS) Access 7.0 and structured with a variety of entities and relationships. These entities include:

• Project: This entity stores the identity data of the Tacis projects including the Contract Number, Project Title, Sector, Sub-sector, Country, Location, EC Task Manager, Project Start & End Date, Status.
• Inception Report: This entity contains the most important information from the Inception Report. It is joined with an "one to one" relation with the Project entity, because there is only one Inception Report in a project's lifetime. This entity also stores data concerning the path and the name of each Inception Report document.
• Monitoring Report: This entity contains the most important information from the Monitoring Report. It is joined with an "one to many" relation with the Project entity, because there may be more than one Monitoring Reports in a project's lifetime. This entity also stores data concerning the path and the name of each Monitoring Report document.
• End of Project Assessment Report: This entity contains the most important information from the Assessment Report. It is joined with an "one to one" relation with the Project entity, because there is only one Assessment Report in a project's lifetime. This entity also stores data concerning the path and the name of each Assessment Report document.
• Comments on Contractor's Report: This entity contains the most important information from the Comments on Contractor's Report. It is joined with an "one to many" relation with the Project entity, because there may be more than one Comments on Contractor's Reports in a project's lifetime. This entity also stores data concerning the path and the name of each Comments on Contractor's Report document.
• Briefing Note: This entity contains the most important information from the Briefing Note. It is joined with an "one to many" relation with the Project entity, because there may be more than one Briefing Notes in a project's lifetime. This entity also stores data concerning the path and the name of each Briefing Note document.
• Kick-off Meeting Report: This entity contains the most important information from the Kick-off Meeting Report. It is joined with an "one to one" relation with the Project entity, because there is only one Kick-off Meeting Report in a project's lifetime. This entity also stores data concerning the path and the name of each Kick-off Meeting Report document.

The IMES database is populated with 3,816 M&E projects and 5,565 M&E reports covering a five-year Technical Assistance period. The following Table presents analytically a description of the content of the database.

Total Number of Projects	2,003
Assessment Reports (ARs)	809
Monitoring Reports (MRs)	2,027
Inception Reports (IRs)	535
Briefing Notes (BNs)	525
Comments on Contractor's Report (CCRs)	1,341
Kick-off Meeting (KOs)	133
Other Reports	195
Total Number of Reports (MS Word files)	5,565

All the reports included in the IMES database were produced between March 1994 and March 1999 and are available in electronic form (MS Word files).

The Local Application

It is the "front end" application, built in MS Visual Basic and used for controlling and managing the stored data. This application enables direct access to the system database through a very friendly user interface. The interface offers an alternative database updating method, combined with great searching capabilities. It also acts as a control agent for the input and output assistants. Analytically, the operations that are supported by the local agent are:

• File Manager capabilities for storing the M&E Reports;
• Searching and browsing the Reports;
• Ability to insert, delete and update the stored data;
• Provision of an interface for querying the database;
• Interaction with MS Office applications in order to provide specific reports;
• Control of the Input and Output Assistants; and,
• Ability to compact and maintain the Database.

Input/Output Assistants

The Input Assistant is a separate tool for loading data from the M&E Reports to the system database. Since all the M&E Reports are written in MS Word templates, Active-X Controls (together with OLE Technology) was used with MS Word to facilitate automatic data entry procedures.

The Output Assistant interacts with MS Office applications in order to provide specific reports, combining data from the M&E Reports. The provided reports are easy to maintain, since they are exported to common office applications, like MS Word and MS Excel. The Output Assistant also includes the Report Generator, which is a tool for preparing statistical reports. The Report Generator was developed in MS Access and provides extended querying and reporting capabilities through an easy interface.

The Intranet Component

The Intranet Component is located on the network Web Server, enabling authorised Internet users to have access to the database. It consists of parameter queries that are executed from common Web pages. These Web pages can be accessed from the network server IP Address, using the http protocol. The VB Script language submits calls to the system database using ODBC driver technologies. The database receives the calls, executes the appropriate queries and exports the results in *.html formats.

Figure 2 below provides an illustration of the IMES Web page.

Figure 2: IMES' presence on the Web

Security

In general the database is isolated and accessible only to the IMES Administrator. There is a "built-in" security mechanism, which protects the stored data and documents. The security system supports specific users groups, and members of these groups are provided differing levels of authorised access. Internet users have read-only access to the database. The official IMES Administrator is the only user allowed to delete data on specific occasions.

Operational Capabilities of IMES

IMES provides authorised Internet users with four important functions or outputs, described in the following table:

Functions	Outputs
Statistics Production	Monthly Statistics for Total Tacis Monthly Statistics per country/sector Cumulative Statistics for Total Tacis List of Problematic Tacis Projects Monitored Monitoring Scores per Indicator for each type of Monitoring Report Monthly Performance Indicators per region, country, sector Cumulative Performance Indicators per region, country, sector
Projects Searching	All Tacis projects implemented in the NIS
Documents Searching & Downloading	All Monitoring Reports (of any kind) produced from March 1994 till March 1999
Documents Browsing & Downloading	All Evaluation Reports and other Reports

Figure 3 below illustrates some IMES functions on the Web.

Figure 3: IMES capabilities via a Web interface

Statistics Production

The system gives the client the opportunity to produce seven categories of standardised statistical reports related to the performance of the Tacis Programme, as well as reports describing the performance of implemented Tacis projects. These reports provide specific performance indicators (monthly or cumulative) per region, country, sector, or status. The values for the indicators are defined by monitoring scores included in the produced M&E reports through appropriate formulas.

Figure 4 below presents an example of one of these reports.

Figure 4: A Statistical report displayed in IMES' Web interface

The above presented statistical report gives valuable information on the implementation of the Tacis Programme in Ukraine in February, 1999. In this case, the average score of the Monthly Performance Indicators ("3,16") shows that there is a success in the implementation of the Tacis Programme in Ukraine in February ("5" means great success and "1" equals catastrophe). On the other hand by studying the Monthly Average Monitoring Scores per Indicator, the client (in this case, the European Commission) can directly discern the phase in which the Tacis Programme faced problems (score <"3") and take corrective measures.

The report known as the "List of Problematic Tacis Projects Monitored" significantly differs from the other reports (see Figure 5).

Figure 5: "List of Problematic Tacis Projects Monitored", a report available with IMES' Web interface

The client can create a list of all of the problematic Tacis projects on a monthly basis (or at a differing frequency), which can be used to resolve problems with specific contractors. Moreover, IMES can produce statistical reports with comparative data related to regions, countries, sectors, allowing a given client to develop specific strategies and launch new projects (see Figure 6).

Figure 6: Cumulative Performance Indicators in IMES

Apart from Commission services, recipients of statistics are:

• Task Managers of SCR A3 & A6;
• The Evaluation Unit; and
• EC Delegations in the region

Searching for Specific Projects

The system provides the client with the capability to search in the IMES database for any Tacis project implemented since 1993, by specifying any of the adopted search criteria (see Figure 7).

Figure 7: IMES' Search Features

Using the search tools to locate a given Tacis project (title, country, sector, start/end date), the client can automatically download all the M&E reports related to this project (see Figure 8).

Figure 8: Search Results

Documents Searching & Downloading

The client is also provided with the capability to search in the IMES database for any kind of monitoring reports produced since March 1994, by specifying any of the adopted search criteria. Moreover, the client can sort the search results by using any of the sorting criteria, such as region, country, sector, report date or title (see Figure 9).

Figure 9: Search Options

Automatically a list of all the relevant reports appears and the client can download them as necessary.

Document Browsing

The system gives the client the capability to browse in a detailed list of evaluation reports produced since 1994 and download them.

Evolution of IMES

The system was originally designed and developed in November 1997. In the first stage of development, the requirements of the clients were analyzed. Two basic questions were posed:

• What are the main requirements set by the client?
• Will these requirements, if properly implemented, result in a system that will meet the needs of the users?

The standard requirements analysis (RA) conceptual framework is presented in the following table:

Goal Requirements	Task Requirements	Functional Requirements	System Requirements
System objectives & Users' needs	Task activities to achieve goal requirements	General system functions for helping users achieve their task requirements	What the system needs to do to implement required tasks and achieve the task and goal requirements

This analysis resulted in the following:

Goal Requirements:

1. Improvement of the Management Reporting on Tacis progress and results.
2. Improvement of the Management of the Monitoring Contracts.
3. Future planning of further Tacis activities in the NIS.

Task Requirements:

1. Study of relevant information on Tacis projects' performance
2. Analysis of overall statistics at NIS level
3. Study of specific information (countries' profiles, important sectors, etc.) in the NIS.

Functional Requirements:

1. Provision of relevant information and data on Tacis projects' performance from the M&E reports
2. Provision of overall statistics at NIS level

System Requirements:

1. Provide the user with the ability to search for Monitoring Reports
2. Provide the user with the ability to search for monitored Tacis projects
3. Produce specific statistical information at NIS level
4. Provide the user with the ability to find Evaluation Reports and Sectoral Reports

With requirements in hand, efforts then concentrated on the design and implementation of the IMES interface. Several fundamental principles, concerning the design of effective interfaces, were taken into consideration:

• Effective interfaces are visually apparent, offering their users a sense of control; users quickly see the breadth of their options, grasp how to achieve their goals and do their work.
• Effective interfaces do not trouble the user with the inner workings of the system.
• Effective applications perform a maximum of work, demanding only a minimum of information from users.
• Interfaces should be user-centered.
• To most users, the interface is the system.

On the other hand, during interface testing different criteria were used, including:

• Visibility of database status
• Match between system and the real world
• User control and freedom
• Aesthetic design
• Consistency
• Efficiency of use
• Readability
• Help and documentation

The performance of the system was tested with the random testing method. Fifteen random real tasks were implemented for every function, proving that the system was quite efficient, reliable and easy to use. System response times are rather low and appear reasonable to the user.

Eventual amendments and improvements have been made, the result of unofficial presentations of the system in Brussels. The system was officially released in September 1998.

Application of IMES

During all the official presentations to the European Commission, the clients stressed their positive experience with the database and its interface. From September 1998 (when all the authorised clients were provided with personal access codes) to March 1999, 619 visits to the system were made by its various clients. Figure 10 describes the number of visits per category of client.

Figure 10: Nature of Access to IMES

It should be stressed that the M-Teams' use of IMES (via the Internet) was limited due to the poor telecommunications infrastructure in the NIS.

The greatest number of visits took place in October 1998, when all the clients had received their personal access codes and tried to navigate in IMES database for the first time.

Current Use of IMES

Today, IMES is at a fully operational mode within a specific organisational framework, providing to all of its authorised users a variety of capabilities. At the start of every month, IMES' System Administrator collects data (M&E reports, etc.) from authorised data suppliers and checks their quality (conformity and consistency to the system) and completeness. This information is then entered into the database of the system. Any problems are resolved by the System Administrator in direct contact with data providers. Only the System Administrator can enter and update the database of the system.

Conclusions

In general, IMES has been evaluated as a very reliable Internet Monitoring & Evaluation Information System. Due to its careful design and successful development, the system has met the needs of its clients and appears to be a very efficient tool to assist decision-making and management. The EC has been very enthusiastic about both the operation of the system and its outputs. Some have characterised IMES as the best information system ever developed for the needs of the European Commission.

About the Authors

Dr. J. Psarras is a professor in the Department of Electrical and Computer Engineering of National Technical University of Athens (NTUA) and Deputy Head of EPU (Energy Policy Unit) Company. He has over 15 years of experience in energy policy analysis, national and regional energy planning, energy and environmental modeling, energy management, decision support and monitoring systems. He has worked as Project Manager and as Senior Consultant in numerous projects funded by EC since 1991.
E-mail: john@epu.ntua.gr

Dr. A. Papakonstantinou is a mechanical engineer with over 15 years of experience in the exploitation of renewable energies, the integration of energy systems in energy grids at a regional level, energy management and waste management, training engineers from energy sector institutions and companies and in monitoring and evaluation.
E-mail: tpapas@epu.ntua.gr

K. Metaxiotis is an electrical & computer engineer and researcher in the Department of Electrical and Computer Engineering of NTUA. He has wide experience in monitoring and evaluation systems, database development, information systems analysis and expert systems design.
E-mail: Kmetax@epu.ntua.gr

Copyright © 1999, First Monday