Who do you Trust in Aid Evaluation?

    It’s a mystery to me how the traditional system of “monitoring and evaluation” of aid projects has persisted for so long. Surely we should have been able to recognize decades ago that having internal personnel evaluate the impact of their organization’s projects is generally a flawed endeavor. In the competitive NGO world where countless organizations are competing for the same limited funds, incentive structures aren’t set up for legitimate internal analysis of impact and results. Donors want proof that their money has produced measurable results, therefore, for the sake of their own institutional survival and the continuation of donor grants, it’s not surprising that many NGOs feel pressured to get “creative” in their evaluations.

    While working for a large international NGO years ago, I witnessed evaluators choosing to proudly report on things such as the number of attendees at regional workshops (statistics that certainly didn’t represent how many of the attendees actually stayed awake during said workshops, let alone benefited from the educational opportunity in any way). But the numbers seemed to please the donors. In the meantime, my scathing memo to senior management about the inappropriate and ineffective field projects I was encountering was completely ignored. What was the purpose of evaluating the projects in the first place if the findings didn’t contribute to meaningful changes in policies and practices?
   
     The perverse incentive structures that are driven in large part by the donor culture tend to force most NGOs to paint a pretty picture of their work. However, this seems like an enormous waste of time and money. I am thrilled to see some groups like MIT’s Poverty Action Lab pioneering more rigorous methodologies to evaluate development projects, but I recognize that these sorts of randomized trials can only be used for quantifiable outcomes and they can be extremely costly to perform.

    So in the absence of a culture of external evaluation or any legitimate “watchdog” group for international aid organizations, what other options do we have when it comes to finding evaluations that we trust? My old stats professor will probably kill me for this, but what ever happened to good old-fashioned gut instincts? Do you trust anecdotal reports by people who have visited field projects and come away with certain perceptions?

     I started considering this concept recently when Marc Maxson at Global Giving introduced me to a new project they’re running that allows people who visit any of their countless field sites to submit “postcards from the field”. These blog-like reports written by non-professionals (mostly by interns and student travelers) have an open-ended framework and are only guided by the question, “what would you tell your friends about this project?” According to Marc, “gut feelings about recommending a project are broad enough to predict deeper problems.” However, as I scrolled through the “postcards” I had a really hard time finding any that were critical of the organizations they highlighted. Are visitors afraid to report on problems or are these organizations actually as perfect as they sound? Marc informed me that, “most volunteers try to self-filter and only say good things publicly, but privately send in negative comments.” Well, it’s not perfect, but at least it’s a start!

    So I have a few questions for the SocialEdge community: 1) Who do you trust when it comes to aid evaluations? 2) Would positive/negative evaluations influence your decision to donate to a particular organization? 3) What is the point of having an entire field of “Monitoring and Evaluation” when the incentive structures of most NGOs are merely set up to tell donors what they want to hear?, and 4) Is there a role for visitor feedback and reviews of aid projects that are based on short-term experiences and gut instincts?

    Tell me what you think!