How We Use Back-of-the-Envelope Calculations in Our Grantmaking
Editor’s note: This article was published under our former name, Open Philanthropy.
Emma Buckland, Program Associate in Farm Animal Welfare, drafts a quick back-of-the-envelope calculation for her colleagues.
At Open Philanthropy, our mission is to help others as much as we can with the resources available to us. When making tough calls about which grants will help the most, we rely on a tool that helps clarify expected impact: back-of-the-envelope calculations, or BOTECs.
BOTECs are rough quantitative models that estimate potential grants’ social return on investment (SROI). Open Phil program staff use them to compare a grant’s expected benefits to its estimated costs.
Despite the name, most BOTECs wouldn’t actually fit on the back of an envelope. They often involve dozens of assumptions, nested calculations, and scenario planning — especially when we’re modeling larger grants or entire focus areas.
Staff across our Global Health and Wellbeing (GHW) focus areas use BOTECs to help answer a key question: Will this grant likely clear our bar for GHW grants?That bar is currently set at a return of slightly over 2,000x in “Open Phil dollars.”
What BOTECs can (and can’t) tell us
BOTECs are flexible tools, and their structure varies depending on the type of grant.
For farm animal welfare (FAW), we might model reductions in suffering.
For movement-building work, we might estimate how much funding an organization could raise for highly cost-effective charities.
Some BOTECs are backward-looking, based on a grantee’s track record. Others are forward-looking, built around a grantee’s theory of change and how likely we think it is to succeed.
To explore the range of possible outcomes, program staff often build multiple versions of a BOTEC — using optimistic and pessimistic inputs to see how sensitive our estimates are to certain parameters. This isn’t about reaching a precise “answer,” but about stress-testing our thinking under different plausible scenarios.
We don’t rely on BOTECs alone to make grant decisions; they are one piece of a broader picture. We also weigh the grantee’s leadership and track record, strategic considerations the BOTEC can’t easily quantify, and potential for “unknown upside” or unusual leverage. Grantmaking is always uncertain, and BOTECs don’t remove that uncertainty — but they do help us navigate it.
Sample BOTECs
Below are four sample BOTECs, simplified a bit for readability. We use pseudonyms to preserve confidentiality, but all draw from real grants we made across vastly different focus areas. We are sharing these examples to give a sense of how we think, where we (frequently!) debate, and how we try to allocate resources to help others as much as we can. As always, we welcome your feedback here.
A recent Phase IIa/b trial tested atorvastatin as an additional treatment for tuberculosis (TB), which kills more than a million people each year. The trial found that atorvastatin helped to reduce TB infection levels more quickly and showed promising improvements in reducing lung damage, based on chest X-rays.
This grant would support a Phase IIc trial to evaluate atorvastatin as an addition to standard TB treatments. The proposed trial would investigate whether different dosing strategies of atorvastatin improve survival rates, accelerate the reduction of TB infection levels, and reduce long-term damage to the lungs. The grant would also support efforts to develop a novel sweat-based diagnostic test for TB.
Benefits of the grant
How many people could benefit?To estimate this, we projected the number of people who would be on drug-susceptible TB treatment by the time results from a future Phase III confirmatory trial would be available. We started with the current number of new cases and made three assumptions: that new cases will decline by 2% per year until the Phase III trial (which we assumed wouldn’t finish until eight years after making this grant), that the current proportion of treated cases would remain constant, and that 96% of cases would be drug-susceptible. This resulted in an estimate of 5.5 million people who might benefit from the treatment.
Speedup:We estimate that funding this trial would result in a one-year speedup of atorvastatin being adopted as a TB treatment.
How effective could the treatment be?Among patients with drug-susceptible TB who are receiving treatment, the current mortality rate is approximately 6%. We do not know how effective this new treatment might be — this is the reason for the trial — but we estimate that a positive result might achieve a 10% relative reduction in mortality (i.e. a 0.6% absolute reduction).
What share of patients would actually adopt this treatment?We estimate that 60% of adult patients would add atorvastatin if it proves effective at reducing TB mortality. This would be a relatively high uptake — higher than the figures we use in most of our therapeutic development BOTECs — but we thought it was justifiable given that atorvastatin is already generic, cheap, and widely available.
Implication for lives saved:Combining the estimates above implies that a successful trial would allow a 0.6% absolute reduction in mortality for 3.3 million people — which equates to preventing roughly 19,900 TB deaths per year, once the intervention has been scaled up.
DALYs:This impact would represent 539,500 DALYs (~19,900 deaths prevented x 27 DALYs per death from TB). This would yield a total value of $54 billion in “Open Phil dollars” (539,500 DALYs x $100,000, the amount of value we assign to averting a DALY).
Total benefit. Since the grant would create one additional year of $54 billion/year, the total value of the grant is 54 billion OP dollars ($OP 54 billion).
Costs:The current trial would cost $320,000. However, it is only a Phase IIc trial, so there would be additional research and rollout costs to consider. We roughly estimate these costs to be $7 million.
Probability of success:We estimate a 40% probability that Phase II and Phase III trials will succeed.
Final SROI:The grant’s SROI is the benefit ($OP 54 billion) divided by the all-in program costs ($7 million), all multiplied by the estimated chance of success (40%). This yields ~3,000x SROI, well above our ~2,000x bar. That is, we expect the grant to generate 3,000 times as much value — in Open Phil dollars — as it costs.
Some limitations of this BOTEC
This BOTEC leaves out some additional benefits — such as the potential value of a new TB diagnostic tool — as well as any improvements to quality of life beyond preventing deaths.
In addition, we only have very rough estimates of what subsequent trials/studies will cost, and we don’t include the possibility that newer TB treatments in development reduce the TB burden beyond the current downward trend.
Updates since making the grant
More than a year has passed since we made this grant, and during that time, we’ve continued to learn about the wider TB landscape. If we were modeling it today, here’s what we’d change about the BOTEC:
In retrospect, we should have assigned a lower chance of success — closer to 30% than 40%.
Conversely, we should have estimated a more dramatic speed-up for atorvastatin adoption — closer to three years than one. We think this is a more appropriate estimate because there weren’t many funding options available for this work, and because we supported the lead researcher at a relatively early stage of his research (before his most promising results were published).
We would also model a longer time period before impact to account for widespread adoption of the drug (rather than just completion of a final clinical trial). We tentatively think widespread adoption might take 18 years, versus 8 years for the clinical trial. The extra 10 years would reduce the grant’s ROI, because we expect TB prevalence to decrease over time. That means a drug widely available in 2042 rather than 2032 wouldn’t have as many potential patients to reach.
While the lower chance of success and longer time to impact would reduce the SROI, the larger speedup would substantially increase it. Taken together, these changes would increase the final SROI.
Current methods for testing lead in paint are resource- and skill-intensive. Portable XRF-based measurement (pXRF) uses a handheld device to quickly and cheaply detect lead in paint and other materials. If successful, the device would allow inspectors to more easily identify lead in paint and prevent it from causing harm.
This grant would fund the nonprofit organization Lead the Way to:
Collect 200-300 painted samples, measured using both pXRF and sophisticated lab techniques (at two universities for additional validation), and develop standard methods for using pXRF.
Put the method into practice by engaging closely with a national government, training local personnel, and using pXRF for measurement.
Test out the method for rapid screening of wet paints.
The key assumptions in the theory of change are:
Lead in paint represents 5% (confidence range: 5-20%) of the overall lead burden in lower- and middle-income countries (LMICs).
For this grant to drive reductions in leaded paint, it needs to increase detection (and through detection, enforcement) by actors using pXRF to test paints with lead levels above regulatory standards.
There is a 50% chance (range: 40-75%) that this grant will validate pXRF enough for it to be published as a reference method.
If pXRF is published as a reference method, we assume that countries representing 20% (range: 10-40%) of the overall burden of lead exposure in LMICs will adopt pXRF to test for lead in paint.
In total, we estimate a 50% chance that this grant affects sources responsible for about 1% of the overall lead burden in LMICs.
(I.e. 5% of lead exposure comes from paint x 20% of countries adopting the method = 1%.)
However, “affecting” doesn’t mean “eliminating.” Even if some countries adopt pXRF, only a fraction of lead exposure from paint will be eliminated in those countries. We’ll need to estimate that fraction:
We assume that half the paint by market volume has high lead levels, and that — if adopted — pXRF will reduce by 20% (range: 10-30%) the formalleaded paint market that accounts for 60% (range: 40-80%) of the lead problem.
Additionally, if pXRF is adopted, we assume it will be used to make a reasonable dent in the informal market. We assume 40% (range: 20-60%) by volume of the leaded paint market is from smaller, informal producers, and that this method could reduce the size of that market by 50% (range: 25-90%).
So overall, we would expect a 20% reduction in the 60% of the market that is formal (12% of the total burden), plus another 50% reduction in the other 40% of the market that is informal (20% of the total), for an overall reduction of 32% (assuming that LMIC countries widely adopt pXRF).
Based on the above calculations, what fraction of lead exposure would we expect this intervention to eliminate?
50% (chance of pXRF becoming a reference method) x 1% (fraction of the global lead burden from paint in countries using the method — the countries make up 20% of the paint burden, which is 5% of the global burden) x [(20% of the 60% market share from the formal sector) + (50% of the remaining 40% from the informal sector)] = .16%
How much benefit will come from a .16% reduction?
We estimate that impacts from scanning will eliminate 0.16% of the total lead burden in LMICs.
We estimate that the total lead burden in LMICs = 117M babies x 5 ug/dL x $OP 3900 per ug/dL = (~$OP 2.3 trillion).
So the total expected benefit is 0.16% of ~$OP 2.3 trillion (2,281,500,000,000), which equals ~$OP 3.6 billion (3,650,400,000).
We also make important but speculative assumptions that (i) pXRF development only deserves 10% (range: 5-20%) of the credit for the total impact of pXRF scanning, and (ii) that the grant speeds up development of the method by 8 years.
Considering these factors, we multiply $OP 3.6 billion by .1 to account for the development share of credit for method use = $OP 360 million.
We then multiply by 8 to account for years of speedup = $OP 2.9 billion.
Finally, we multiply by .8 again to account for Open Phil’s share of credit for development = $OP 2.3 billion.
The expected value of this grant is ($OP 2.3 billion in benefit) / (total cost of project [~$350,000]) = ~6,500x SROI in expectation. For any one parameter, changing the best guess to the more pessimistic end of the confidence range still yields an SROI above the ~2,000x bar.
Supporting a fundraising organization for effective charities
Giving Ground is the only professional organization promoting effective giving in its country. With just 3.5 full-time employees, it raises money almost exclusively for GiveWell’s top charities.
In many cases, Open Philanthropy program staff use backward-looking BOTECs to estimate future impact — so to assess a potential new grant, we created a BOTEC based on Giving Ground’s past performance:
With ~$300,000 in operating expenses, Giving Ground raised ~$800,000 for GiveWell’s top charities over roughly the last 2.5 years.
We think that most of this sum is directly attributable to Giving Ground. A survey of Giving Ground donors suggested that 15% of the money allocated came from people who had already encountered effective giving in other contexts, so we applied a 15% reduction to the funds that Giving Ground raised when we created our estimate.
After the 15% adjustment, our best guess is that over the next two years, Giving Ground will raise $2.27 for GiveWell for every $1 it spends: $800,000 x (1-.15 [counterfactuality discount]) / ($300,000 [operating expenses]) = 2.27x. Our bar for effective giving interventions is a 2x ratio, so this clears our bar.[ref] For giving opportunities that act as money multipliers, we typically look for a return of twice our funding (even though this conceptually is higher than our normal ROI bar since we think effective giving is >0.5 times as effective as our bar). The extent of this higher bar is subjective, but we believe a higher bar is warranted for a few reasons:
We think the way we model “meta” funding opportunities, which are more distant from impact, is more likely to overestimate cost-effectiveness by failing to account for additional efforts required by other actors to achieve impact. (If we give money directly to GiveWell, we’re responsible for 100% of that. If our grantee spends $1,000 and raises $2,000 for GiveWell, our $1,000 may be only partly responsible — for example, the donors reached by the grantee may also have been influenced by GiveWell’s own advertising, or by efforts from other actors in the effective giving world.)
Our model is likely to overestimate the impact from additional donations to “high-impact” charities because it assumes that any funding attributed to the grantee would otherwise not have gone to charity at all (rather than going to a less impactful charity).
Intuitively, we aren’t excited about supporting opportunities that could spend an additional $1 to generate only slightly more than that for high-impact charities, even if all donations are counterfactual. Supporting these opportunities would mean that small errors in our calculations (or unforeseen expenses) could result in negative impact (e.g. spending $1 to raise $0.90), and potentially send us into a meta trap.[/ref]
Improving conditions for broiler chickens
This grant falls within our focus area of farm animal welfare.Note that the FAW team uses a different bar than the rest of our GHW focus areas.
Feather Forward runs national campaigns to convince corporations to improve their animal welfare policies. The campaigns focus on stopping the use of caged hens to produce eggs and on improving the welfare of chickens raised for meat (called “broiler chickens”).
Just like the grant above, this is a backward-looking BOTEC. When considering a new grant to Feather Forward, we modeled their impact under their previous grant as follows:
Costs: $1 million — Feather Forward’s total expenses across the period the grant covered (two years).
Animals impacted: We estimated that Feather Forward impacted 34 million animals alive at any time through a combination of:
13 new country-level cage-free commitments from companies.
Two major global cage-free commitments to which Feather Forward contributed.
Six country-level broiler welfare commitments, including five major retailers.
Implementation progress of cage-free commitments in their country.
Credit adjustment: We gave Feather Forward the following credit for each set of wins:
/p>
80% of the credit for national cage-free and broiler wins: they were the major group behind those; the remaining 20% of credit goes to another organization in the country.
10% of the credit for global cage-free wins: the organization directed almost all of the campaigns in their part of the world, which comprised roughly 10% of wins globally.
Commitment/implementation effort adjustment: In most cases, after animal protection groups secure a corporate pledge, they need to get the pledging company to follow through on their commitment.
For new commitments, we apply an adjustment (usually 40-60%) to account for the cost of subsequent work.
When we credit groups for companies’ progress on implementing commitments, we similarly apply an adjustment to give credit for the work of securing the commitment in the first place.
Given these adjustments, we estimated that Feather Forward’s work improved the welfare of 5.4 million animals alive at any time. To determine the full value of this impact, we apply additional considerations that include:
How many years these campaigns accelerated adoption of reforms
The degree of welfare improvement per animal
The extent to which each chicken can suffer
After incorporating these factors, we calculated that this grant easily exceeded our internal bar for farm animal welfare interventions. While our FAW BOTECs incorporate different moral weights and welfare improvement assessments than our other focus areas, the core approach remains consistent.
Note: The Open Philanthropy Project was formerly known as GiveWell Labs. Before the launch of the Open Philanthropy Project Blog, this post appeared on the GiveWell Blog. Uses of “we” and “our” in the below post may refer to the Open Philanthropy Project or to GiveWell as an organization. Additional...
Editor’s note: This article was published under our former name, The Open Philanthropy Project. Some content may be outdated. You can see our latest writing here. This is a writeup of a shallow investigation, a brief look at an area that we use to decide how to prioritize further research....
Some diseases that kill many people receive very few resources for the development of diagnostics, treatments, and vaccines. In relative terms, R&D funding varies by 10 times or more across diseases, often without good reason. This suggests that science funders could find high-impact opportunities by targeting the most neglected diseases.
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
Cookie
Duration
Description
cookielawinfo-checkbox-analytics
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional
11 months
The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance
11 months
This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy
11 months
The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
