The World - Latest Comments in Discussion: How Do You Know a Good Charity When You See It?

Re: Discussion: How Do You Know a Good Charity When You See It?

Angilee Shah — Thu, 15 Aug 2013 11:24:24 -0000

Iqbal, is there a way for individual donors to put this kind of evaluation in practice?

Re: Discussion: How Do You Know a Good Charity When You See It?

Charity Navigator — Fri, 09 Aug 2013 09:27:38 -0000

Since we signed the letter to the donors of America about the 'overhead myth,' I'm sure you can guess where we stand on this topic. We believe financial measures are part of the mix, but aren't the only thing donors should examine. Our President & CEO's blog explains it very well: http://ht.ly/nMjqi.
But as we endeavor to rate charities based on their results, we find that very few charities actually measure and publicly report on those results. We need funders of all kinds to provide far more resources for the charities themselves to measure and report on their results. Without such an investment, most charities will never be able to know for certain if their efforts are as effective as they should be and as a consequence precious resources will be wasted.

Re: Discussion: How Do You Know a Good Charity When You See It?

Charity Navigator — Fri, 09 Aug 2013 09:21:48 -0000

The fact that this conversation is taking place is wonderful. Too many donors still give only with their heart. It is encouraging to see more donors interested in learning about the impact of their gift and willing to do a little work to decipher which charity is really delivering its mission.
We believe donors/ social investors should examine 3 aspects of a charity's performance before giving: the charity's Financial Health, Accountability & Transparency, Results Reporting. We offer more details on these tips here: http://ht.ly/nMiwp and http://ht.ly/nMiFu .

Re: Discussion: How Do You Know a Good Charity When You See It?

mike — Fri, 09 Aug 2013 04:05:11 -0000

Thanks all for the replies here, very insightful. Many thanks!

Re: Discussion: How Do You Know a Good Charity When You See It?

shamsa birik — Thu, 08 Aug 2013 19:14:25 -0000

Thank you Amy for hosting this great talk glad to be here for the first time. As we ask this question of ''How do you know a good charity'' the organization I work for is in the process of downscalling most of the support staff due to declining donor funding and shrinking internal financial reserves. The organization explained that there is less funding being allocated to organizations working in the Horn of Africa as donor attention is now more focused in Syria or Mali and the financial meltdown in Europe has really affected things so they can no longer retain most of the staff. The process of terminating contracts has been painful to both the organization and the staff most of whom have served the organization more than 5 years. The organization has a good management team, good in program implementation, has good relations with the community they work with, so what exactly are they doing wrong to have their proposals eliminated during call for proposals? Simple they have been doing the same thing they did 20 years ago no change, no innovation, no sustainability to even retain their staff. My worry was not the minimal funding being allocated by donors to the Horn or the crisis in Europe but rather my question was 'what is this organization doing to adjust to these realities'. To me there is real crisis when charities refuse to innovate they simply are applying same fundraising strategies they did 20 years ago, they been have caught up by time. Learning from this mistake is key, the organization should invest an actually spend more money instead of downscalling to reap the fruits later. For example they need to hire a fundraising officer, invest in a communications department, hire a quality assurance officer, use crowdsourcing strategies to fundraise, run hackathons to help look for effective solutions to social problems, have better internal audits, more strict financial systems, use participatory video communication as an M&E tool, conduct research and surveys that will better help in interventions, use social media and simply get a new website that is more interactive that demonstrate what they do. Also enhance better cooperation between academia and international development as well as they should find the link between social enterprise and development. to get enough financial reserves. Most charities are faced by the same shocks from time to time and get vulnerable, do charities safety nets? yes. To me a good charity is one which does most of these, uses new technologies and hires people who are passionate about development. I would use the example of Kenya Red Cross which was once a 'dying' organization like ours but now is one of the best Charities in Kenya (https://www.kenyaredcross.o...

Re: Discussion: How Do You Know a Good Charity When You See It?

David Baron — Thu, 08 Aug 2013 18:56:42 -0000

It's good to know that so much thought has gone into how one avoids such problems. Thanks for the primer!

Re: Discussion: How Do You Know a Good Charity When You See It?

David Baron — Thu, 08 Aug 2013 18:53:50 -0000

Very helpful. It's good to know that you keep an eye on these possible negative impacts.

Re: Discussion: How Do You Know a Good Charity When You See It?

notjanetjackson — Thu, 08 Aug 2013 17:01:08 -0000

Hi Iqbal, thank you very much for your informative reply. Do you think you could talk a little bit more about your last statement -- that sometimes, it is not possible or even desirable to measure the outcomes of a charity program? Why would this not be desirable?

Additionally, what proportion of charities would you say practice rigorous, controlled evaluations? Is it true that impact evaluation is becoming more widely regarded as a necessity in assessing the success or failure of different programs? How can we ensure that all charities incorporate it?

One last question -- are long-term assessments part of impact evaluations? Or is the time frame usually more short term, and if so, what implications does that have for the validity of the evaluation?

Thanks so much!
Janet Li

Re: Discussion: How Do You Know a Good Charity When You See It?

Holden Karnofsky — Thu, 08 Aug 2013 16:28:12 -0000

David, I think it's usually possible to assess the risk on a case-by-case basis. Many of the charities that work on public health work *with* the government; for example, the Against Malaria Foundation consults with national malaria control programs to find areas that can't be covered (with insecticide-treated nets) with existing funds, and then steps in and funds distributions to fill the gaps. On a case-by-case basis, it's usually possible to look at what a government's budget and activities are in a particular area and determine whether the charity you're supporting is supplementing, adding on, or potentially getting in the way.

Each charity review we do has a "possible negative and offsetting impacts" section in which we discuss what we see as the most relevant risks.

Re: Discussion: How Do You Know a Good Charity When You See It?

Dayna Brown — Thu, 08 Aug 2013 16:27:57 -0000

David and Amy, thanks for organizing this and for facilitating an interesting discussion. In answer to your question, David, it is the responsibility of charities and any outside actor coming in to "do good" to be sure they are not undermining local capacities or disincentivizing local actors--whether they be the goverment, civil society or the private sector. We should "do no harm" in the long-term, even with humanitarian aid, and pay much more attention to the HOW, not just the WHAT we provide. As Iqbal points out, it requires and forward planning and coordination. Charities must also have an understanding of what already exists, why it is or isn't working and an agreed approach with local actors to strengthen what exists and to build capacities for what is needed. Aid agencies should be thinking from day one how they will work themselves out of a job, identifying with local actors what their roles and responsiblities are and setting clear benchmarks and indicators for success. I often question the effectiveness of organizations that have been working in the same place for decades, so ask they why they are still there. What are they doing now that builds on what they and others have done in the past? What are they doing to work themselves out of a job?

Re: Discussion: How Do You Know a Good Charity When You See It?

Iqbal Dhaliwal — Thu, 08 Aug 2013 16:20:53 -0000

The main challenge to a standardized system of evaluating charities (or any development programs) is (a) the tendency to focus on outputs not outcomes, and (b) not being able to distinguish good evidence used to evaluate one program from the not so good evidence being touted by another organization.

(a) Output vs. Outcome: While it is true as Joel said that "charities are engaged in very different activities and employing very different theories of change", this variation is often much lesser in terms of outcomes that they are trying to affect. For instance, charities could be trying to supply different types of school inputs (black boards, chalks, textbooks, new buildings), or instead be focusing on improving teaching quality (training courses, pedagogy) or quantity (to reduce student-teacher ratio), or trying to increase demand for schooling among parents and kids (scholarships, free uniforms, free school based lunch/meals) or teach to the right level (remedial education for the weaker students instead of teaching just the syllabus) - yet despite the large number of theories of change that these interventions are based on, most of them will have the common goal of either increasing student attendance or improving learning outcomes (ability to read or do math). So if we can shift the way we evaluate charities from looking at outputs (number of books delivered, schools built, teachers trained, textbooks supplied etc.) to outcomes (attendance or learning), then we have a far better way of comparing organizations.

(b) All evidence is not equally rigorous: Even when organizations try to measure outcomes, they often do it in a way that makes it wrong to attribute causation (see my earlier discussion in response to Amy's opening question on the problems with widely used measures like before-after, or difference across seemingly similar groups or the use of stand-alone anecdotes). There are other much more rigorous ways to measure the true impact of a program - randomized evaluations are one of those, especially in situations where a large amount of resources are to be committed to a new program or a change in policy.

So a big challenge is to have more implementers and donors focus on outcomes of their work instead of outputs, and try to measure these outcomes in a rigorous way (where possible or desirable - in some cases neither is true).

Re: Discussion: How Do You Know a Good Charity When You See It?

Iqbal Dhaliwal — Thu, 08 Aug 2013 15:47:19 -0000

David, this has been a great conversation, so thanks for organizing this. In response to your question, if the charity is a small organization with limited ambitions (will only work in one district or a few villages), then this is not really a problem because large NGOs or governments will mostly not change their general policies or efforts. But if this is a charity that has broader ambitions for expansion and feels its success may lead to pull back by others, then weaving in an impact evaluation into its roll-out plans will help answer your question. For instance by comparing the number of "institutional" deliveries or immunization rate for the ENTIRE population in the treatment villages vs. comparison villages (which continue to get existing services but no new programs), you will know what is the "net impact" of the charity that will account for the "pull-out" by existing organizations. The charity can then assess whether the cost and effort of the INCREMENTAL institutional deliveries or immunization rate is worth their effort.

But it does not need to even come to this calculus with a bit of planning - charities can work in partnership with other large agencies in that region to coordinate their efforts so that instead of crowding out efforts, charities can focus on addressing different parts of the puzzle - so one example would be for the charity to raise awareness of the importance of institutional delivery and mobilize women to go to pre-natal clinics while the government can focus on improving the staffing in those clinics instead of sending half the nurses into the field on awareness campaigns. This requires more coordination than just rolling out your own program but the right thing to do in places where your concern is a salient issue.

Re: Discussion: How Do You Know a Good Charity When You See It?

David Baron — Thu, 08 Aug 2013 14:59:52 -0000

I work with Amy Costello here at PRI's The World. Many thanks to our guests for taking questions! I'm curious to know: How can you evaluate if a charity -- even if it's effective, say, at saving lives -- is undermining local efforts to solve the same problem? For instance, if an outside charity steps in and provides obstetrical care to women in some rural part of Africa, doesn't that reduce the incentive for the local government to provide that service? How can a charity avoid this unintended outcome?

Re: Discussion: How Do You Know a Good Charity When You See It?

Holden Karnofsky — Thu, 08 Aug 2013 14:44:10 -0000

Amy, that's an accurate characterization of GiveWell. We started as a group of donors trying to decide where to give, and we try to serve effective altruists trying to do as much good as possible, so our focus is definitely on asking the question "Where should we give?" - and giving as thoroughly analyzed and clearly discussed an answer as possible - rather than on providing a large number of ratings. We hope that in the process of doing deep, thorough analysis in a transparent way, we'll help others think about the more general principles and approaches that could be used to evaluate a larger number of charities for people with different values.

Re: Discussion: How Do You Know a Good Charity When You See It?

Holden Karnofsky — Thu, 08 Aug 2013 14:42:55 -0000

You might want to check out our 6 tips for giving like a pro. At the moment, the main things we look for in a charity are evidence of impact, cost-effectiveness/bang-for-the-buck, and "room for more funding" (ability to use more donations to accomplish more good).

Re: Discussion: How Do You Know a Good Charity When You See It?

Holden Karnofsky — Thu, 08 Aug 2013 14:34:33 -0000

Brian, I agree, and I think this viewpoint is gaining currency (see www.overheadmyth.com). For our part, we've never used financial statements as anything but a relatively small piece of the picture. To us, the important metric is something more like "good accomplished per dollar" than "percentage of money spent in a particular accounting category." Understanding the former means reviewing evidence of impact, seeing a charity's work in the field, etc. - much more than scanning the financial statement. (More at our criteria)

Re: Discussion: How Do You Know a Good Charity When You See It?

Holden Karnofsky — Thu, 08 Aug 2013 14:33:08 -0000

Mike, I couldn't agree more. One of the things that GiveWell is trying to do is create a more mature conversation about charity, based on numbers and claims that hold up, even if they don't paint as rosy a picture as fundraising brochures. We write publicly about our reservations with our top charities; we post updates on their progress that include details on the good, the bad, and the ugly; and we publicly discuss our own mistakes as well. More:

* 2012 discussion of a former top-rated charity that had run into some major hurdles, and a guest post on our blog from the organization.
* Report on our current #1 charity that includes our reservations and uncertainty points.
* List of GiveWell's own shortcomings as an organization.
* We have also written about the need for an attitude change more generally - see, for example, our proposal to reward failure.

We're trying to flip the script so that charities that give a detailed, substantive, nuanced picture are more likely to be rewarded, whereas charities that stick to fundraising narratives that don't tell the whole story are not.

Re: Discussion: How Do You Know a Good Charity When You See It?

Holden Karnofsky — Thu, 08 Aug 2013 14:29:47 -0000

I haven't seen any such measures that I consider to be convincing to an outsider. I agree that community ownership and empowerment are very important, but I haven't found a way to be confident that an organization is bringing them about. Improvements in health and wealth are easier to measure, and I think these capture forms of empowerment (albeit at the individual level).

Re: Discussion: How Do You Know a Good Charity When You See It?

Holden Karnofsky — Thu, 08 Aug 2013 14:28:36 -0000

Amy, it definitely takes more work to gain confidence in even a single charity than most people have time to do. For all of our top charities, we've reviewed the academic and other evidence behind what they're doing, reviewed large amounts of internal documents on their activities, budgets, processes, monitoring data, and funding needs, spent substantial time in person with the people running the organization, and performed extended visits to see their work in the field - and we still have large numbers of unanswered questions. For donors who don't have time for that sort of investigation, I'd recommend our 6 tips for giving like a pro (or, of course, simply donating to our top charities).

Re: Discussion: How Do You Know a Good Charity When You See It?

Lilly Dimling — Thu, 08 Aug 2013 14:16:52 -0000

I understand what you are saying, why provide a something if the community has not identified it as a need? Whose responsibility it it to find out what it is the community thinks it needs? It seems that charities/aid orgs would not have the funds to do this AND the programming itself. And each aid org has a specialty, like poverty alleviation or education. One org can not address all the needs of a community I presume.

Re: Discussion: How Do You Know a Good Charity When You See It?

Iqbal Dhaliwal — Thu, 08 Aug 2013 13:21:02 -0000

Actually there are many completed and ongoing randomized evaluations that try to measure the impact of community participation and empowerment on improving development outcomes, and to understand what are the mechanisms that make some of these interventions more effective than others. On the face of it, nothing seems more obvious - involving and empowering communities in local development programs should lead to better outcomes. But how effective such community involvement is in improving development outcomes can depend a lot on the context as well as the way such involvement is structured (more details in the reply I just posted above in response to Eric's question).

Re: Discussion: How Do You Know a Good Charity When You See It?

Iqbal Dhaliwal — Thu, 08 Aug 2013 13:15:28 -0000

Issues of community participation and empowerment are among the most fascinating questions in development. On the face of it, nothing seems more obvious - involving and empowering communities in local development programs should lead to better outcomes. But how effective such community involvement is in improving development outcomes can depend a lot on the context as well as the way such involvement is structured (details below).

Many of J-PAL’s affiliated professors have investigated the impact of greater community participation on the effectiveness of development programs, not just to learn whether it works, but to identify ways to improve the effectiveness of community participation and community ownership as strategies for making the programs more effective. So far the results are mixed: In Uganda, community participation in monitoring the quality of local healthcare both improved the quality of care and health, even reducing child mortality ( https://www.povertyactionla... ). On the other hand, in India, mobilizing community members to monitor local schools did not encourage effective participation in school governance ( https://www.povertyactionla... ).

In Indonesia, top-down audits were more effective at reducing corruption in road construction projects than community participation, but this was likely because this particular type of corruption was difficult for community members to observe ( http://www.povertyactionlab... ). Yet in another evaluation in Indonesia, community participation led to substantially higher satisfaction with government social assistance programs ( http://www.povertyactionlab... ).

So while we are beginning to understand the mechanisms that make community participation more effective, there is still more investigation that needs to be done, which is why we have so many ongoing impact evaluations on these very questions ( https://www.povertyactionla... ).

Re: Discussion: How Do You Know a Good Charity When You See It?

Holden Karnofsky — Thu, 08 Aug 2013 12:18:38 -0000

Thanks Amy! I am the co-founder of GiveWell, which aims to find outstanding giving opportunities and publish the full details of our analysis to help donors decide where to give. We've spent years looking for charities that donors can give to and be confident of their impact, and we've found it to be a huge struggle. I think the biggest misunderstanding out there is simply overestimating how much is known about effective aid. Many people imagine that as long as the charity they're supporting is well-intentioned, honest, isn't spending too much on overhead & fundraising, etc., that their money is well spent and accomplishing good. In reality, we know very little about how most aid programs affect the people they're trying to help. It's hard to collect good data, it's hard to generalize from data, and it's hard even for someone in the field to really connect with and understand people with radically different cultures and living conditions. We've tried to find the "easiest" bets for donors - cases in which we can answer nearly every question might ask - and even for these (our top charities), there are huge numbers of unanswered questions.

That isn't to say the challenge isn't worthwhile. I believe that aid does a great deal of good in aggregate, and the track record of health interventions in particular is strong. But assessing the isolated impact of a particular organization is very difficult, and there aren't established reliable methods for doing so.

Re: Discussion: How Do You Know a Good Charity When You See It?

Brian Mittendorf — Thu, 08 Aug 2013 12:04:45 -0000

I would be curious to hear the panelists' views on the ideal role of financial statements in evaluating charities.

First, my two cents...

For some reason, the popular view now is to see things as a financial metric vs. impact metric comparison. I would say they are better viewed as complements in that prudent financial practices are a necessary but not sufficient condition for a charity to be "effective". Further, the current practice of plugging in a few numbers into a formula to evaluate financial statements may be quick and easy but is often counterproductive. A full understanding of the financial picture of a nonprofit requires just as much care as gaining a full understanding of their practices. For some reason, many seem to be unwilling or unable to make that investment so instead opt to discredit financials as a useful source of information (e.g., referring to the "overhead myth").

Re: Discussion: How Do You Know a Good Charity When You See It?

Lilly Dimling — Thu, 08 Aug 2013 11:53:51 -0000

The title of this event is "How Do You Know a Good Charity When You see It?". I would love to panelists to answer that question. As an individual donor, what should I look for, what questions should be asked directly to the nonprofit that could inform the donation decision?