This fund (3F) was established in order to facilitate and accelerate the development of the theory, practice, and profession of evaluation. It is based on the belief that explicit attention to critical study of the methodology and foundations of a discipline and its applications—especially in the case of an emerging or radically changing discipline—can avoid many dead ends in its development, and nurture valuable new perspectives. These improvements, in the case of evaluation, are not merely academic refinements, because the whole operation of society depends critically on the careful identification and demonstration of the success or failure of its efforts at improvement and its response to crises, i.e., on ethical professional evaluation. Funding from 3F is allocated with significant weight given to social payoffs from evaluation for those in need, including unconventional needs for which a case can be made.
Evaluation itself had to spend fifty years to achieve a moderate degree of legitimacy after its discard, at the beginning of the twentieth century, into the trash heap of scientifically untouchable topics, a rejection based on a superficial and fallacious critique of evaluation by the positivists. That indefensible blunder meant that many, perhaps most, social scientists turned their backs for half a century on full frontal attacks on the great problems faced by our global society, a decision which almost certainly cost us all dearly. 3F is an attempt to reduce the chances of similar mistakes and similar costs. Its modest resources are devoted to supporting proposals for new approaches to the task of extending the domain of rational scientific efforts at objective analysis of evaluative issues at either the applied or theoretical level, including methodological or foundational issues.
In this effort, 3F’s aim is to generate the development of new perspectives on and applications of evaluation, with some preference to ‘out of the box’ or ‘long shot’ projects and proposals that are likely to find funding hard to get from the usual sources because of their departure from the current research paradigms and/or their relatively low chances of success. Comments, from a number of Nobelers amongst others, have supported the need for this kind of approach as filling a gap in the current research funding portfolio across all research disciplines, reaching far beyond the sciences, e.g., into jurisprudence, technology, mathematics, history, literature, and the classical arts.
Goals. The 3F approach is intended to be, and should be seen as, simply complementary to the great efforts made by the American Evaluation Association (AEA) and similar professional organizations world-wide. In particular, while the AEA covers a vast range of support for professional evaluation members and their needs, including an excellent system of honorary awards for research and service, our intent is more narrowly focused, specifically research intended:
(i) to generate new research rather than reward completed research;
(ii) to focus on a certain sub-area of research (which we might call normative meta-research), which is just a small but crucial part of the big field of research on evaluation itself;
(iii) to facilitate the emergence of new or massively transformed paradigms, critiques, and practices, rather than the mere refinement of existing ones, even excellent ones; and
(iv) to emphasize continued consideration of the potential social benefits of refining evaluation theory, practice, and methodology.
Structure. 3F is set up as a donor advised fund under the umbrella of the Marin Community Foundation, the third largest community foundation in the US (in US legal terminology it is a nonprofit 501(c)(3) organization). That management and funding is set up to continue in perpetuity, subject to the continued supervision of the 3F Advisory Committee, a group of experienced evaluation specialists and disciplinary leaders representing many evaluation approaches, including many ex-presidents and editors of evaluation organizations and publications such as our leading journals and anthologies (their names and interests can be found at the 3F website FasterForwardFund.org; Jane Davidson is our Chief Operations Officer).
3F Annual Grants—2016 (note new deadline—November 15):
First, remember that every entry must be eligible as a “3F Project”. To get a better understanding of possible eligibility, we will post online a few projects funded last year. (After completion, we will post short summaries of 3F funded grant projects with links to more extensive summaries or the actual completed projects, as appropriate on our website.) This year, in conjunction with our regular General Topic grants, we have added 3F Focal Topic Grants as possible areas for exploration (described in more detail below). For further clarification of what a 3F General Mission project might be, here are some ideas that illustrate the possible topical range of a 3F project:
- examining the evaluative assumptions and methodology used by historians of warfare and warfare technology in supporting evaluative judgments;
- the scope and limitations of ‘big data’ methodology for social policy analysis;
- supporting work by a small international specialist online group focusing on specific technical problems in intercultural cooperation and evaluative comparisons;
- the application of game theory (or positive psychology; or addiction theory) to the ethics of suicide prevention;
- the quality of a sample of evaluations done by a state’s legislative analysts (or by a federal inspector-general’s office; or by the World Bank); i.e. meta-evaluation;
- the utility in certain circumstances of what might be called ‘televal’ by analogy with the now highly active field of telemedicine, e.g., to bring technical or management skills from highly experienced evaluators to Nepal or Namibia or the Northern Sioux;
- developing new curricula and pedagogy for teaching evaluation methodology in the K-16 realm (or in the professional development area);
- the validity of quality-adjusted life years (QALYS) as the emerging key metric for national/international social (or medical) interventions, or an analogical problem in educational research;
- simplified public access/participatory (P.A.P.) evaluation systems such as Emily’s List, Amazon reviews, JD Powers, US News & World Report (and see Foci list).
In general, supported proposals will undertake to produce new understanding or evaluative information about, or methods for doing, evaluation, in a form suitable for immediate publication, typically in a paper, chapter, or a set of these; or perhaps using augmented reality video technology.
Meta-evaluations are of course eligible (see bullet 5 above), but support depends on their methodological significance, e.g., to improve the logic of rubric formulation.
What’s not typically funded. It is unlikely that funding will provide support for purchase of travel or durable equipment, since online video and technology rental are usually adequate; and of course, work that has been previously published, online or in hard copy, will not be eligible. Also ineligible is any evaluation of the usual applied kind that currently occurs in the score of fields where thousands of professional evaluators already work, not even if it uses some novel methodology; the research must be on that novel methodology, not just embodying it.
Funding amounts. The financial limitations on 3F support at the moment mean that applications for a few thousand dollars are perhaps slightly more likely to be supported than those requesting or requiring tens of thousands (partly because there can be more of the former); and requiring hundreds of thousands puts a proposal beyond the current 3F range. This year’s funds available will be in total, approximately double those provided last year, but individual grants will most still be about the same (i.e. mostly small; under 10K). We can be useful as co-sponsors for larger (and smaller) projects since that approach means many proposals can be funded for a lesser amount than requested due to our financial limitations. Running big evaluations is often expensive; inventing valuable new methods or critiques is often very important and not so expensive.
2016 Proposals to 3F (See FasterForwardFund.org for more details and to download an application, due midnight, November 15, 2016.)
The current requirements are fairly simple:
(i) describe what you and your group/organization is and wants to do, when, where, and how. Include methods to be used, materials and services required and a timeline. (Timeline must note a proposed mid-point task(s) and/or indicators and a due date for them; as well as a completion target date, usually less than 12 months.);
(ii) describe indicators you think should be used to assess progress and conclusions of the project and note these in your timeline;
(iii) provide an argument, and perhaps evidence (e.g., a needs assessment), that this effort is consistent with the aims of 3F. This should include a discussion of social payoff and epistemological benefits (i.e. knowledge + understanding/insights & behavior);
(iv) give your estimate (and evidence) of feasibility and importance, with emphasis on expected tangible results that will be widely available; include qualifications of the personnel involved (note expertise, experience, knowledge, training, etc.).
(v) submit a detailed budget and, if appropriate, evidence for its accuracy and cost-effectiveness (e.g., don’t include travel when video can do the job). Note that 3F will not pay overhead beyond about 10%, preferably less.
(vi) keep all this down to a maximum of a 3-5,000 thousand words (e.g. by severely condensing your cv), to reduce the investment of time by you and us. (It is acceptable to point to a website which has additional information about you or your project. However, the website information will not necessarily be read, so the submitted proposal must contain sufficient information to be complete and stand on its own.)
Budget suggestions. There are two considerations to keep in mind that distinguish 3F from some–perhaps most–other funding agencies. First, the reviewers, other things being equal (i.e., assuming equal probability of success and equal importance of a successful outcome), will be likely to fund less expensive projects over more expensive ones, since they are trying to maximize the payoff for the discipline and society from a limited budget of their own. So the common practice, in proposal budgeting, of multiplying your normal salary rate by the number of days or weeks you think the work will take, may make you uncompetitive with someone, whose project is otherwise equally promising, who sets their payment at the minimum level that will get them to take on the task. Second, reviewers will not proportionately penalize long-shot projects by comparison with those whose probability of success is higher: they will give substantially greater weight to the payoff compared to the expectation of success.
Proposal deadline. Proposals for this round of funding are due by November 15th, 2016, but earlier submission is recommended, since a few smaller projects may be funded before that date, and others may be considered more extensively, and doing this spreads the rating load for our readers.
While we strongly recommend using colleagues for extra input and critique of your proposal, they should not be identified as co-authors for that kind of input. A principal investigator (or at most 2) and financial point person, must be identified to take responsibility for the project and its funds.
Financials. Payment to successful applicants is best done via a legally qualified philanthropy, e.g., your university or organization (keeping in mind that 3F will normally expect to pay no more that 10% overhead, and see FasterForwardFund.org for more details and for overseas funding requirements). Payment will normally be in three installments: a small sum for start-up costs; a mid-project payment ; and a final payment (after completion of project and acceptance of final report, including a 150-200 word summary). Reporting 3F results openly, which we think is important, means that a summary of your project and perhaps a link to your final project may be posted on the web and perhaps elsewhere.
There is no hidden agenda in the sense of a preferred approach other than that outlined here (for a copy of our current rubric visit FasterForwardFund.org/. . . ). The actual judging for awards, and all other strategic management decisions, is overseen by the 3F Advisory Committee. Decision time will depend on the number of entries, but we’ll try to keep it between 4-6 weeks.
Feedback. Evaluators should practice what they preach, and there are good reasons for that practice, so we welcome suggestions for improving this proposal or its presuppositions. Send them to our email: [email protected] with the subject: Suggestions.
New 3F Project-The Faster Forward Fund Special Focus Prizes (4F):
I. The Essay Prizes. A new $1,000 “Special Focus Award” will be given for each of the five categories below (assuming minimal standards are met). Each of the five following topics will be run as a separate competition.
Written entries must be submitted with less than 1,500 words, with no names or other identifying information in the body of your submission. Your name and contact information must be included on a separate cover sheet and in the body of your email, which should have the subject line: 3F Foci # (insert focus number); no cv or nonprofit affiliation necessary.
Send entries to: [email protected] by midnight PST on 15 November, 2016 (an extended deadline).
Special Focus 1 - Ethics & Scientific Method
Do reason and scientific method oblige us to adopt and report our evaluations from an ethical point of view; or do they oblige us to be skeptical about or to ignore ethical considerations? Note: this means evaluating the ethical defensibility of the evaluand itself, not the, or not just the, methodology and creation of the evaluation. In other words is ethics simply one branch of applied science? (A 2,500 year old problem; time to solve it?)
Special Focus 2 – Peter Singer Prize
Best design for the evaluation of at least three charity watchdog organizations; e.g., Charity Navigator. These outfits are the primary evaluators used by donors of millions of dollars every year. Which of them gives the best advice is therefore a really important question and 3F should try to provide a good meta-evaluation. Note: problems of data access and transparency of results should be addressed, and a cost estimate included. An effort should be made to provide for a low cost (less than 25K) and a higher cost (50K) approach. You can also indicate your interest in doing the evaluation (with further funding) as yes or no.
Special Focus 3 – Meta-evaluation Checklists
Best critique of available meta-evaluation checklists, including at least those by Dan Stufflebeam, GAO, and Michael Scriven. Extra points will be awarded for the best improved version based on or unlike the published ones (in other words, a prize for the best invention and use of a meta-meta-evaluation checklist).
Special Focus 4 – Meta-evaluation of Mainstream Science
At long last, serious work has begun on investigating whether the hard-core sciences are in fact scientific, at centers in Stanford and the University of Virginia. Their first reports are out and both are very critical. The first meta-meta-evaluation is also out and is very critical of UVA’s effort. Can we improve this area of research?
Special Focus 5 – Future Topic Ideas
Suggest and support a special focus 3F topic or topics for 2017. What would be useful to accelerate the discipline or practice of evaluation (i.e. take us faster forward)?
Why is it important? What problems is it causing because it is not being adequately addressed?
I. Project Support Awards. If you would like to actually attempt to address one of these issues, please submit it as a proposal for regular 3F General Mission funding rather than as an idea or approach for someone else to address next year. (See earlier requirements listed and visit FasterForwardFund.org for complete details.)
A slight preference will be given to 2016 general grant applications that tackle one of the foci on this list—this means if two submissions of equal merit are entered, and only one can be funded, preference will be given to a 2016 special foci topic submission.
Thank you for your interest and please visit FasterForwardFund.org to find out more about our other projects too!