14 Steps in Constructing Results-Based Monitoring and Evaluation Data System

Replies to This Discussion

Permalink Reply by Steve Metzger on November 21, 2007 at 10:37am

Nimit,
Welcome to the new site. I checked out the PacPlan website. Very impressive. MSI has started offering Information management systems solutions for M&E. We don't offer any "products" in the traditional sense, but we do create custom reporting systems for USG Agencies (USAID and State mostly). How much customization takes place on a client to client basis?

Steve

• ▶ Reply

Permalink Reply by Nimit Leelasorn on November 22, 2007 at 12:14am

Steve,
Our system solution is the fast track approach and was designed for clients to apply the customisation themselves. In most cases we provide the initial consultation and allow for further and on going customisation to be continued by clients or local consultant. The customisation effort could start from simple M&E process handled by an individual M&E specialist or be used by the entire organisation, either small or large, sophisticated organisation/Country/Region. The customisation effort could take from few days to a few months for most sophisticated M&E process in large organisation/Country'Region. The purpose of the solutions system is to move the clients out from theory of M&E system capacity building to practice as quickly as possible to work and resolve their real organisation M&E issues.

Nimit

• ▶ Reply

Permalink Reply by Claude Kakule on February 7, 2008 at 9:38am

Dear Mimit

Thank you for sharing with us your 14 steps in constructing a results-based M&E Data System. In general, M&E data system doesn't get more attention from experts responsible of designing the results-oriented M&E System.

One of our key concerns in developing countries actually is data availability and quality of data. An other concern is the actual use of available data/information. In this regard, could you please share with us your experience on how to make sure that, following the above mentioned 14 steps, the organization/country can collect useful data/information, with a minimum level of data manipulation. Regarding the quality assurance mechanisms, it is sometime very easy to say then to put in place, especially when the M&E budget is small.

An other big challenge in developing countries is the data transmission, mainly from remote areas/villages. Not only the internet connection may be a problem, but it may be also very difficult to get a driver (private or public) who can take a diskette from one area to the head office. The mobile phone may be used also to transmit data from remote areas in some countries, but the implement of this approach has also some limitations: quantity of data, need for data validation at some levels between the remote area /office and the head office, etc. Are there any lessons learned regarding this issue?

Last but not least, in many organizations, M&E work in general, and data collection and processing, in particular, is still seeing by many staff, including top managers, as an additional task. While designing a results-oriented M&E system, the question is always how to make sure that this is not the case. Any thoughts?

I am sorry to reply a bit late to this very interesting topic.

Best regards.

Claude

• ▶ Reply

Permalink Reply by Nimit Leelasorn on February 18, 2008 at 7:49am

Dear Claude,
Sorry for the long answer to the question as this topic is very important. It needs a lot of explanation and illustration after gaining interest from the initial reading of the 14th Step.

I must thank you for well constructed questions of the 14Step since the publication of the article.

Congratulation for discovering the missing gap in constructing results based M&E data system.

Thank you for sharing with us your 14 steps in constructing a results-based M&E Data System. In general, M&E data system doesn't get more attention from experts responsible of designing the results-oriented M&E System.
The little attention is surprising considering that, after all, the data is an integral element of the M&E data system. Well designed data and structure would allow M&E to provide the management with information they require from the system for informed decision making. But from our experience, it is also unfair to expect attention to M&E Data system from those who have not experienced this environment before. Really, it is a catch 22 situation. We were very lucky with our M&E project in early 1990 where the champion was very concerned about the data up front and got us involved in their M&E initiative. Now, many years down the track, this organization has achieved previously unimaginable results considering limited resources and funding and was able to adapt and manage various crises, which is not common, particularly in the government organizations in developing country.

One of our key concerns in developing countries actually is data availability and quality of data. An other concern is the actual use of available data/information.

In this regard, could you please share with us your experience on how to make sure that, following the above mentioned 14 steps, the organization/country can collect useful data/information, with a minimum level of data manipulation.
Key approach in ensuring data availability, quality and usage of the data, must be a focus on the data being transformed from the existing environment and stakeholders. Stakeholders must collect data that they need for them to perform their duty. (i.e. Project Manager must collect the activity indicator and monitor progress in time to determine the success of activity in the project). This way, he must ensure the data is collected on time and accurately. However, the M&E data system data structure design must cater for possibility of cross referencing of the data, comparing to similar district or activity as part of performance comparison. This would ensure the bad data is identified.

There are various data validation techniques (i.e. check sum…) that could be applied to the information system but most important in data accuracy is its usage by stakeholders. Once stakeholders have confidence in the data from the data system then the data quality and usability would gradually improve and ensure the M&E data system is sustained for the long term.

Regarding the quality assurance mechanisms, it is sometime very easy to say then to put in place, especially when the M&E budget is small.

Referring to our first client again - their budget was very small too (as it was unheard of any government organization in developing country to have a large budget for M&E data system as that has to go through various government processes, even though at the same time there are large funds allocated to tangible items, like car, van, truck or hardware, if not all the way to parliament, depending on the level of funding required). M&E data system could be implemented cheaply, if it adopted the top down approach where M&E data system is implemented at the head office and all the data reported manually in the data input form (replacing the current project manual report) and faxed (or forwarded by any other low cost and basic method) to the head office for data entry.

The report data could provide top level view of organization and policy to the decision makers as well as individual project level monitoring, where the M&E data system report could be sent back to individual project or district office. This may sound cumbersome but this is a huge progress already compared to the most current environment where efforts were committed in the reporting but yet did not enable monitoring or measuring both policy and project level effectively. Achievements in area of M&E data system should be compared with their own environment or the environment with similar capacity and limitation, not with the highly developed environment/countries or more advanced environment. It can be used as the target but keeping in mind that this advanced environment in developed country did not just happen overnight. If one can accept this reality, the progression on a long road of M&E data system development can begin.

An other big challenge in developing countries is the data transmission, mainly from remote areas/villages.
Before the MIS system, information (manual report) was passed from remote village to central government or head office via post or courier (or whatever other means, for hand delivery or phone reporting). If this manual reporting process does not exist, before embarking on the M&E Data system construction, the current manual capacity building must take place first, or happen in parallel, with emphasis on organisational and human resources capacity. Some M&E data system tools have this capacity of building capability and could speed up the process. The same approach could be used with M&E data with basic available technology (i.e.: diskette and USB drive, these days). In early 1990 when we commenced our first M&E data system for the government in developing country, personal computer was readily available, although limited in number, but internet/electronic data communication lines were not available due to budgeting limitation as well as accessibility to the technology in remote locations. The data was filled in the prepared computerised data input form in (or via observation, etc.) in the remote areas and mailed by post or courier or hand delivered to the provincial office (or head office), where it was entered at the provincial office for regional consolidation. This keyed data was passed onto the head office (to avoid duplication and data key error). This transmission used diskette as the only means available at the time but the challenge was to get this process in place. Once it was established, the data flowed smoothly. The same approach was used for data dissemination, where computer reporting was provided from the central office to the remote locations. This process of using existing resources with only some small adjustment was affordable and worked well.

After a few years the organisation was able to secure large budget due to tangible results, to acquire better data transmission technology. This also included the established data system process. This approach allowed the government with limited resources to focus on one issue at a time - first data flow and second - technology upgrade. This is how it happened in developed countries in early years, when development was pioneered. They didn’t suddenly jump into the high technology environment.

Not only the internet connection may be a problem, but it may be also very difficult to get a driver (private or public) who can take a diskette from one area to the head office.
If the organization already has the process of manual reporting in place, this diskette (or USB drive these days) is another media to use in the existing process. If there is no process in place, it should be established first, considering basic options first, rather than upgrading the internet or data communication infrastructure. Please note, a well paid driver in developing country who is eager to do the job would still be much cheaper than setting up a reliable communication infrastructure for transmitting data on the internet. If the driver issues can’t be resolved, implementing technology for the data transmission would be thousand times harder. This could be a good test for management capacity.

Let’s not forget how things were before computer and internet. All government organizations used to send their reports by mail and courier Phone call (or observation) - that has been done from the beginning of time and they know this well.

To transform from the basic manual reporting to data reporting as well as implementing data communication technology (internet) infrastructure at the same time and with still limited capacity, could lead to very little achievement and no one really knowing is technology or data causing the issues. A few years later the same process would start again as the older people are transferred and the new people with bright ideas but no real experience come along. This challenge would continue well into the future.

Consideration must be made for the volume of data (refer to 14 steps) as this would provide justification whether to use internet connection or diskette sent by mail/courier from one area to other (this may take 2-3 days). It should be remembered here that for any government to produce a report could take years. I have seen the government official in a developing country very busy publishing base line data and when I inquired, I found that the data was 2-3 years old. In this light, 2-3 days to transfer is good already.

Developing countries must use caution in aspiring to high level of technical sophistication while embracing technology, and rather get back to the basics of Data management using the most basic form possible. Personal computer, not even server use, as they have to create foundations (at low cost), and then, once this is established, gradually introduce more sophisticated technology. This would allow the M&E specialist to understand their need and enable him/her to instruct the technician on what technology they want. But, at the same time, from the first go, the PC could produce information for high level management already. Do not forget, the high level management only want report of the progress and results.

We were asked by General Director from one of the government organization in a developing country why can't we just get someone to key data at the head office and give him the report, why do we spend 6-9 months trying to resolve data transmission technical problem. (as people didn't really understand the technology)

The mobile phone may be used also to transmit data from remote areas in some countries, but the implement of this approach has also some limitations: quantity of data, need for data validation at some levels between the remote area /office and the head office, etc. Are there any lessons learned regarding this issue?

Let’s consider the simple form of transmitting the data from remote areas to central office, before the sophisticated data communication technology, which has been in practice for many years (mail or government courier, or fax, or Phone call). Using input form faxed to the head office approach would allow the data originator to input the data themselves. The quality may not be very good , but at the initial stage of implementing M&E data system it would be a start to ensure that all the data from remote locations can be aggregated to the country or policy level. This would also encourage the data usage at the high level management and identify the bad data as well as using the good one, but correction or improvement of the data availability, quality and usability would come from the top level of organisation (not the other way round like in developed countries). Most top level management in government organisations in developing countries with long experience in organisation would recognise data that does or does not make sense.

Second approach in providing the M&E data system to the remote area would be to enable the data entry as well as reporting facility to assist the local monitoring as well as usage. This data could be transmitted to the head office in a most simple form (diskette) and mailed or couriered to the head office for country or policy consolidation. Yes, it could take a few days for data to get to the head office and may seem cumbersome but at least this would avoid duplication of data entry and introducing possible data error. The best data entry option would be to have it done at the originating location as any error could be resolved locally. (Please keep in mind M&E data reporting, prior to the data system, could have been done once or twice a year at best due to capacity limitation, and only after using the M&E data system for some time, could progress to quarterly or monthly reports, so 2-3 days of transmitting the data to head office is fast already).

Once this process is established, all stakeholders (remote area management and central head office management) would be using data from the system through reporting. Then it could be easily justified to upgrade the technology for say, internet. By this time the originating location would know exactly what information and speed will they get for the investment in more sophisticated technology.

Please note technology does not ensure better data management. The ability to use the information itself is the key to success from the very beginning. Then the technology can be applied as well as resources in the organisation to support this technology, but all this takes time to establish within government process.

Last but not least, in many organizations, M&E work in general, and data collection and processing, in particular, is still seeing by many staff, including top managers, as an additional task. While designing a results-oriented M&E system, the question is always how to make sure that this is not the case. Any thoughts?

In designing M&E data system, data collection and processing must be an integral part of the organisation’s work and process flow, not a stand alone task. This way some duplication reporting could be eliminated. Two most effective approaches in pursuing organisational support, is to secure strong backing from an influential actor, as even top managers do also report to someone higher, i.e. politician has influence over a number of high level decision making processes of the civil servants, and pressure from the top would definitely ensure support. Another approach is to introduce an incentive (i.e. cash award or position progression). This is very effective as the staff and top manager could see the benefit within a short term i.e." the best of the best will get reward". It is extremely important during the initial implementation to gain support from all stakeholders using whatever means possible as most staff don't see the long term benefit of M&E data system, so it needs to be translated into short term goal and target. This incentive approach could be adapted and applied, as suitable, to various cultures and regions.

M&E data system should not be seen as only indicator data collection for performance, it should also facilitate the performance process, i.e.: Monitoring and Evaluation of personal performance would also ensure the hard work performance by staff or top management as they would also get reward at the end. M&E data system must serve all tasks and processes around performance monitoring and evaluation.

P.S. Another thing to consider - when it comes to constructing M&E data system, most people seem to be very concerned about the needs and wants of individuals or organisation or stakeholders, and pursue that direction. From experience - those needs and wants could be addressed in M&E data system, and if there was one or two individuals or organisation or stakeholder those needs and wants would be simple and manageable but, when constructing M&E data system in public sector, there may not be one or two participants, this number could multiply into hundreds or even thousands. An M&E data system that tries to accommodate all needs and wants would become very complex MIS indeed and consequently hard and very expensive to implement (not possible in developing country due to capacity limitation) as well as long term sustainability would be in jeopardy.

In looking at the data and its structure, we walk away from serving the need and want of individuals but construct the data system to suit fundamentals of organisation's core objectives and environment (policy/plan, organisational, institutional, and local and stakeholder objectives) with the top down implementation, working towards those goals / objectives. Interestingly, this data structure would continue and gradually serve those individual needs and wants as M&E data system progressively evolves in the organisation.

Building M&E data system is a long term process, which is best to introduce progressively to provide progress benefit as it evolves, i.e.: first 6 months - provide policy/plan, project, stakeholder monitoring; 12-18 months - provide outcome monitoring and policy/plan and stakeholder evaluation; 24-48 months - provide monitoring and evaluation at grass root level......etc).

Thank you again for you valuable contribution to the discussion.

I am sorry to reply a bit late to this very interesting topic.

In summary: availability, quality and usage of the data is depends on the people in the organisation. The trick is how to get this chain reaction going (not better technology or data system tools alone, or larger budget). Basic suitable data system tools could help facilitate and fast track the transformation process.

M&E Data management is a progressive transformation of the data users and organisation themselves. You are very right about using the term “put in place” I will make sure that in the future I will use something like transformation, conversion….etc.

Thank you again for well constructed questions.

Nimit

• ▶ Reply