Call for your Input: Assessing Multilateral Organisation Effectiveness

Replies to This Discussion

Permalink Reply by Tamirat Yacob on January 29, 2008 at 10:38am

Dear Sheila,

Thanks for the document you have posted hear and you have given us the chance to comment on it.
I will be back to you with comments and suggestions that i would have.

with many thanks,

Tamirat

• ▶ Reply

Permalink Reply by Claude Kakule on February 7, 2008 at 7:24am

Dear Sheila

Thank you for sharing this very exciting document with us. I have gone through it and I have some preliminary comments/suggestions. As a starting point, I must confess that I have never used the Balanced Scorecard in my work; I know only the theoritical aspects. In this regard, some of my comments may not be appropriate; I would be sorry for that.

a) General comments:

1. The Balanced Scorecard is an interesting approach and performance management system that can enable organisations not only to assess its present performance, but can also try to capture information about how well the organization is positioned to perform in the future. As the starting point of the MOPAN BSC is the four quadrants (why not six ?), it would be appropriate to seek additional views on the suggested quadrants during consultations with other perspective partners. In this respect, national governements ("customers"), civil society and private sector should have their says.

2. While designing and implementing the BSC approach, one should not underestimate the fact that people tend to get what they measure. As Kleiner claimed, what comes first in every institution is keeping the Core Group satisfied. In this regard, clear measures and mechanims should be in place in order to reduce the level of "data cooking" at each level of the process.

3. I personnaly agree with the recommandation made by the group suggesting that Methodology # 2 should be used as a starting point coupled with the Implementation Option # 1. However, Implementation Option #1 should be also viewed as a starting point; partners would move to the Implementation Option # 2 when possible. In additon, the concept of "without jeoparadising the credibility or public image of the MOPAN" should be clarified. Some concerns raised when implementing the so called "Independent" evaluation of programs/projects or "independent" humanitarian needs assessments may be raised again at the end of the BSC process.

b) Specific comments

1. On page 11, table 2: for more credibility, I would suggest that data collection administration under Methodology # 2 be conducted by the survey consultant, with the support and collaboration of national consultants and the lead MOPAN donor in country.

2. On page 17, I can read "The findings and/or score on each KPI will be an amalgam of the data received from the different data sources which are identified....". This is logic, but my question is how to make it happen in reality when we all know that data availability and data quality are main concerns both of the national governements and donors ? How to deal with big discrepancies in data (from different sources) taking into consideration that some questions/indicators would be very sensitive?

3. MOPAN BSC rating system (page 17): the suggested approach is very interesting, especially when we know that when a project is rating "Satisfactory" by the World Bank, for example, the meaning is quit different if the same concept was used by GTZ ! While I can understanding why the four quadrants are weighted equally (25%), I don't understand why the rating of each dimension within each quadrant is based on the number of KPIs. The number of KPIs is very volatile; one can pass from 45 KPIs to 34 after futher discussions (ref. page 2). I would suggest that the weight of dimensions in each quadrant be calculted indirectly.

4. Consultation and piloting approach (page 19): I suggest also that Civil society (including Communities of Practices on MfDR) and private sector should be part of the consultation process.

5. Recommandations # 6 (page 23): The timeframe seems to be very ambitous. Consultations with MOs and national governements may require more time then expected.

I may come up with additional comments before or during the meeting.

Best regards.

Claude Kakule

• ▶ Reply

Permalink Reply by Seydou Yaye on February 24, 2008 at 6:55am

Bonjour Sheila
J'ai lu avec intérêt le document que vous nous aviez soumis pour commentaire. Désolé pour mon niveau d'anglais ce qui limite ma compréhension et mes observations de fonds. Permettez moi de remercier le groupe de travail pour cette initiative trés attendues dans nos pays qui cherchent à la fois à améliorer la performance de l'aide et réduire les nombreuses missions des partenaires techniques et financiers. Une estimation commune (évaluation conjointe, supervision commune) des différents apports (internes et externes) permettront sans nul doute d'augmenter l'efficacité de l'aide et donc d'améliorer l'impact des opérations.

J'apprécie également les propositions de méthodologies et les options d'implantation du processus proposées par le Groupe. J'adhère parfaitement à la 2ème méthodologie qui peut ,aprés essai, conduire à la 3ème. Il est évident que l'environnement des pays et les capacités des gouvernements et des représentants locaux des organismes de développement compteront beaucoup dans l'application de cette méthodologie. Elle me paraît être la meilleure méthodologie.

Pour ce qui est des options d'implantation, l'option 2 m'apparaît plus intéressante puisqu'elle se situe dans la durabilité. Il est admis en général que l'utilisation de consultants pour ce genre d'activité n'est pas durable puisque les connaissance et les expériences ne sont pas toujours transférées. C'était le maillon manquant de l'aide au développement. On a fait recours systèmatiquement à l'expertise externe au détriment de l'expertise interne (tant au niveau des Etats qu'au niveau des organismes de développement) ce qui n'a pas permis de renforcer durablement les capacités des pays en développement. Le groupe a proposé la première option, mais je pense qu'il faille l'améliorer.

• ▶ Reply

Permalink Reply by Abdou Karim Lo on February 24, 2008 at 12:08pm

Bonjour Sheila,
Merci de nous avoir donné l'opportunité de nous prononcer sur cette importante initiative dont l'objectif est de définir un modèle d'évaluation de la performance relative à l'aide au développement. La multiplicité des modèles et des critères d'évaluation ne permettent pas effectivement d'avoir une appréciation partagée de l'efficacité de l'aide.

Aussi, l'harmonisation des critères d'évaluation est-elle effectivement nécessaire à la fois pour les partenaires au développement, les Etats bénéficiaires de l'aide et les gestionnaires de programme. Mais, faudrait-il qu'ils aient tous la même perception et la même compréhension des critères et indicateurs à utiliser.

En ce qui concerne la méthodologie de MOPLAN, j'estime que la deuxième approche est la meilleure. En effet, elle favorise une démarche participative et se situe dans une approche de "parties prenantes" (stakeholders). Si l'étude est confiée à un consultant, les pays membres et les gestionnaires de l'initiative ne seront plus dans une perspective de management de la connaissance (knowledge management).

En d'autres termes, il convient d'impliquer dans la mise en œuvre de l'étude, les membres de chaque antenne pays, à côté de consultants dont les compétences sont avérées. Cette approche prendrait moins de temps, couterait moins chère et permettrait de dégager les leçons apprises.

Nous soutenons le projet et j'estime que le Cop MfDR Afrique a un rôle prépondérant à jouer dans sa mise en oeuvre. Nous sommes prêts au Sénégal à investir toutes nos forces pour un succès éclatant de cette initiative.

S'il existe une version française du rapport méthodologique de MOPLAN, il faut le faire parvenir aux francophones.

Abdou Karim LO, Expert en organisation
Sénégal,

• ▶ Reply

Permalink Reply by Zaam Ssali on February 25, 2008 at 9:08am

Dear Sheila,

Please receive my comments on the MOPAN paper as follows:

a) There is a risk of confusing this assessment with the Paris Declaration Country Evaluations. Clarity of both initiatives will be important for all stakeholders as well as the timing for the two their analysis and the value addition for the two (PD and MOPAN).

b) I suggest that implementation follow option #2 because there is an advantage in using consultant resources from outside and the staff from within the MOs, MOPAN members and Secretariat in that it may allow for quicker implementation. The internal staff have the insider knowledge while the consultants will provide the expertise and hopefully they will have more time for this task while the internal staff provide the internal details.

c) Regarding consultations at country level; I don't know how it works in other countries but in Uganda we have donor groupings around sectors and areas of expertise. This could provide a good entry point for consulting donors in specific countries. These groupings work closely with the National governments and PRSP coordinating units.

d) I have noted that the MOPAN members are few at the moment, how then will we ensure that this assessment is owned by all?

e) I agree with the position proposed in the paper of further consultations with MOs and a sample of National Governments for refinement of proposed methodologies is key for the success of this initiative.This will hopefully be useful in tackling the issue raised in d above.

f) The time-frame seems very ambitious. more time may be required for consultations with MOs and National Governments.

regards,

Zaam

• ▶ Reply

Permalink Reply by Tamirat Yacob on February 25, 2008 at 9:37am

Dear Sheila,

Thanks for the shared document. I have tried my best to go through it. Actually some of my observations are very similar to Claude Kakule’s and I do agree with his comments. But I would like also to put my contribution here and ask for more clarification too.

1. On the implementation section, what is the difference between the Application of the approaches on the proposed methodologies of #2 and #3 that are presented in Table 2, 3 and 4 (page 11-13)?

2. On page 23, the time frame set for the piloting of the survey on National Government is only two full months (if it is to be started on the beginning of July and end at the last day of September). To be more realistic I think we have to give them at least a maximum of three months.

3. On Page 27, as it is indicated in the Annex B, Methodology #2, it uses “Country Partners” (I think it is to mean Civil Societies and Private Sector) as a source of Data for the evaluation. But I couldn’t see where they fit in as there is no questionnaire deigned for them, no consultation mechanism, etc. So need some clarification.

4. Concerning ODA, we have the KPI#9, 11, 12, 22 and 24. But none of the questions, in my humble opinion, in these KPI clearly tries to investigate, for example, how much share of the ODA is taken by the “Technical Assistance” and how much is “for development financing”, etc. And I think this will also be one of the concerns of the ODA recipient countries. So it needs some attention when designing the questionnaire.

Appologies for being so late and responding at the dead line.

Best regards,

Tamirat Yacob.

• ▶ Reply

Permalink Reply by Devendra Parsad Ruhee on February 25, 2008 at 2:38pm

Dear Sheila,
I have gone through the content of the MOPAN Options Paper. It is a very good background paper which will allow participants in the meeting to engage in discussions as to the best strategy for achieving the goals and objectives of MfDR.
I wish to make the following general observations/comments on the Options Paper:

Methodology Options: Implementation of Option#2( mixed approach) as a starting point seems to me to offer the best opportunities for African countries as it will encompass national governments' perspectives in the process.It will enhance organizational learning, better results-based management and reporting thus increasing the scope for greater collaboration and involvement of partner countries.
Development of a common approach and harmonisation of processes around the proposed MOPAN Balanced Scorecard methodology for assessing institutional effectiveness will no doubt bring about a more objective assessment of aid effectiveness of donor-driven national development projects and programmes.However, we would need to be briefed on the performance and micro-indicators.

Collection and analysis of secondary data will no doubt play a pivotal role in the assessment of developmental effectiveness under knowledge management.Performance information at corporate and country levels, as mentioned in the Options Paper, would greatly facilitate assessment of development effectiveness.
Implementation methology proposed under Option#2 and assessment will include all shareholder views. I note with great interest that the methodology will take on board national government and country partners as another primary data source.
For assessment of institutional performance, RBM tools will be used in pursuit of improved organizational efficiency and effectiveness at delivering development results.
Consultations which, I hope, will include civil society will emphasize the organizational learning and capacity-building benefits of participation in an RBM Organizational Assessment.

Best regards.
Dev Ruhee

• ▶ Reply