The AfCoP Retrospective Evaluation Is Released!

To support the transition of the African Community of Practice on Managing for Development Results (AfCoP-MfDR) to the African Development Bank in early 2012, the AfCoP Core Management Team and Secretariat commissioned an evaluation of the AfCoP’s performance including its relevance and effectiveness to date. The aim of this exercise is to carry out a retrospective evaluation of the AfCoP as a whole – based on its objectives set out in 2007 with a focus on its usefulness to members, its impact on the ground, and identifying strategic gaps to be filled in its second phase.

This work has been led in a participatory manner by the international development evaluator, Michele Tarsilla, and is now available for download here. We invite you to read it and provide any comments. Below is a summary of the methodology and findings.

---------------------------------------------------- 

Scope and Methodology. The evaluation was conducted from March to June 2011. It used several data collection methods, including a document and specialized literature review, two online surveys, semi-structured interviews with a wide-range of stakeholders, and a thematic analysis of MfDR related discussions held on the AfCoP’s website. The evaluation paid specific attention to whether the AfCoP achieved its initial objective of increasing individual MfDR knowledge and the usefulness of the online platform in facilitating South-South exchanges. The evaluation also explored country ownership of MfDR and the AfCoP’s impact on national development processes such as Poverty Reduction Strategy Papers (PRSPs), National Development Strategies, as well as planning and budgeting, monitoring and evaluation (M&E), statistics work, capacity building, and accountability and partnership mechanisms.

Key Findings

AfCoP membership has led to an increase in individual MfDR knowledge. As a result of participation in the AfCoP, survey respondents acknowledged an improvement in their understanding of all the five MfDR pillars (see Table 1). Furthermore, 52 percent of respondents claim that they have adopted MfDR concepts and tools that they discovered on the AfCoP website. The majority of respondents (88 percent) claimed that they are now motivated to modify their practice or work methods because of their exchanges on the AfCoP website. Furthermore, 38% of respondents indicated that they had contributed to their country’s MfDR process since joining the AfCoP.

The AfCoP has contributed to the adoption of MfDR at the country level however challenges still remain in raising awareness and securing high level government support. Overall, 54 percent respondents claimed to have seen some improvements in their country planning processes over the last three years. Among them, 70 percent claimed that the AfCoP provided a substantial contribution to this improvement (in particular due to the knowledge gained from online discussions). The following items were listed by respondents as improvement linked to AfCoP participation: enhanced focus on results at all levels (39%); introduction of MfDR good practices in key national sectors (36%); strengthened in-country MfDR capacity building programs (29%); increased number of National CoPs (29%); increased accountability and inclusiveness in national planning processes (16%); increased use of MfDR tools and methodologies (8%); better MfDR exchanges among national key players (6%); increased focus on results during PRSP formulation processes (5%); and increased use of performance-based budgeting (5%).

It is however, difficult to attribute AfCoP membership to improved national planning. Although impact on national planning and budgeting processes were not part of the original mandate of the AfCoP, it is encouraging that members stated that the AfCoP has an influence on national events and processes, a desk review of PRSPs demonstrated that it is difficult to demonstrate causal linkages between AfCoP membership and the creation of results oriented documents and processes. Furthermore, only 4 percent of AfCoP members surveyed stated that they played a role in informing the development of their country’s PRSP. This tenuous links between AfCoP membership and the results orientation of national planning documents may warrant a separate and more in depth review of the results components of PRSPs.

Existence of National Communities of Practice (CoPs) plays a role in building MfDR capacity and high-level buy-in at the country level. Nearly half of survey respondents belong to an AfCoP National CoP. These respondents acknowledge that existence of a National MfDR CoP significantly improves the engagement of high-level government officials. This is because they help draw attention to the importance of results-based planning and development. Since 2009, eight National CoPs have been established in Burkina Faso, Côte d’Ivoire, the Democratic Republic of Congo, Kenya, Mali, Niger, Senegal, and Zimbabwe with the direct support of African Development Bank (AfDB). The CMT’s National CoP team conducted an informal internal review of its activities to date. One of the main findings of this review was that National CoPs have conducted MfDR training of approximately 3,000 National CoP members. However, National CoPs seem to be lacking resources to fully implement their actions. Thus success of National CoPs therefore depends on the level of commitment and engagement of individual members as well as availability of resources at the country level. It may be interesting to follow up on these findings in a subsequent independent and formal evaluation (an in depth assessment of National CoPs and their impact on the national planning process was beyond the scope of this evaluation).

AfCoP members comprise of a wide range of results practitioners and stakeholders from Africa and around the world. The AfCoP network represents a valuable resource in helping to move the MfDR agenda further. As of 2010, 40 percent the members were civil servants working in planning, monitoring and evaluation, and budget; 21 percent were from civil society organizations and the private sector, and 18 percent work for development agencies, mostly in field offices. Over 80 percent of the AfCoP membership base is African and the majority of members are from sub-Saharan Africa, mainly West and East Africa. This network of senior MfDR practitioners offers a useful audience and target group for expanding MfDR implementation across Africa. One of the indirect benefits of AfCoP has been that it also attracted members from other regions (20 percent), thus improving cross-regional exchange of knowledge (Brazil, Singapore, and Malaysia have been particularly active on the AfCoP website and Annual Meetings).

The AfCoP website is a valuable tool for increasing MfDR knowledge and facilitating networking, despite initial concerns about connectivity and access in Africa. Overall, 60 percent of respondents claim to have developed some contacts with other AfCoP members and gained knowledge. Furthermore, 72 percent of respondents report that AfCoP online resources are relevant to their needs, but only one out of ten members has been an active contributor (while the remaining 89 percent are considered onlookers). A large number of respondents reported that they did not actively contribute to online discussion or blogs out of fear that their candid comments on the state of MfDR within their organization would make them loose their job (despite an anonymous posting function available on the website). On the whole, survey respondents report that the AfCoP website is easy to navigate and that they are able to locate the information they need quickly: the majority of respondents (63 percent) find the website easily accessible.

The organic development and semi-informal nature of the AfCoP have allowed the community to continually respond to the changing needs of its members. This flexibility was an important feature of the AfCoP’s success as the community grew from 20 initial members at its creation in 2007 to over 2,000 members in early 2011. Having a dynamic and responsive Core Management Team and a Secretariat is a significant factor in ensuring the continued success and relevance of the AfCoP.

Uncertainty about future funding, the recent launch of certain aspects of AfCoP implementation (such as National CoPs), and the flexibility of the initial AfCoP framework makes it harder to determine the longer term impact of the AfCoP’s work. Despite the value of having a dynamic and responsive community, the AfCoP lacked a concrete theory of change. Although partly justified by AfCoP’s innovative and experimental nature, the lack of both, a well-articulated theory of change and M&E system represents a program limitation. In future, the AfCoP should give itself a stronger Monitoring and Evaluation Framework with tangible baselines and goals/targets to achieve over the next few years. The possibility of having sustainable long term funding provided by the African Development Bank should offer a way to ensure and measure longer term impact of the AfCoP on the ground.

Conclusion

Overall, the evaluation found that the AfCoP achieved its initial goals. The most impressive finding is the significant increase in MfDR knowledge of individual AfCoP members. As mentioned above, the first phase of the AfCoP focused on building a South-South capacity building community that increased individual MfDR knowledge. To that extent, the AfCoP has been successful. Timing is opportune for the AfCoP to move beyond individual knowledge and create a groundswell of support for MfDR at the national level. It is currently understood that the African Development Bank will provide funding for the AfCoP going forward, focusing on building linkages between AfCoP member countries and regional economic communities tied to achieving the MDGs. This new direction will help further build MfDR capacity at the country level. In addition, the AfCoP needs to link closely to national planning process and should focus on engaging high-level government officials to ensure that good MfDR practices are adopted. Finally, this evaluation represents the very first attempt to evaluate a regional CoP-MfDR. Other regional CoP-MfDRs may also be interested in evaluating their performance. This would allow for an easier comparability of findings and would be particularly beneficial both operationally (i.e. identifying good practices in the design and management of regional CoPs and politically/intellectually (i.e. assessing the strengths and weaknesses of each regional CoP-MfDR) thus enabling more effective south-south cooperation based on respective comparative advantages.

We invite you to read the AfCoP Retrospective Evaluation and provide your comments below.