Dia-blogging: Evaluators Can’t Do It Alone; Need to Build New Partnerships

Riitta Oksanen is the Vice-President of the EES. She works as a senior advisor at the Finnish Ministry for Foreign Affairs, Development Evaluation Unit. In this post Riitta diablogues with Rakesh Mohan who is the Director of the Office of Performance Evaluations (OPE), an independent and nonpartisan agency of the Idaho State Legislature. He is a former board member of the American Evaluation Association (AEA) and currently serves on its Evaluation Policy Task Force. Riitta and Rakesh started discussing the topic of building partnerships in evaluation when they first met at the 2014 AEA conference in Denver.

Key Points

  1. Evaluation is a service function. It only becomes meaningful when it meets the information demands and when the results are used.
  2. Independence in evaluation becomes a curse unless complemented with strong partnerships that ensure relevance and responsiveness.
  3. Because evaluations involve making judgment about prioritization, distribution, and use of resources, evaluations will always be inherently political.

Riitta: Rakesh, in Denver, you talked about building new partnerships to promote evaluation use among policymakers. In order to accomplish that, you said evaluators would need to broaden their discussions at conferences beyond evaluation methods and results. What did you mean by that?

Rakesh: At the closing plenary session of the 2013 AEA conference, I talked about the 4 Ps that are missing in discussions at AEA conferences. These 4 Ps are policymakers[1], politics, the public, and the press.

Rarely do we see policymakers at AEA conferences. Of more than 3,000 attendees at the AEA conference in Denver this year, I believe there was only one policymaker – I had invited him to participate in a panel discussion. Furthermore, the politics of evaluation is seldom discussed at AEA conferences. There seems to be a myth among many evaluators that evaluation has nothing to do with politics. They believe they need to stay away from politics in order to maintain their credibility as independent and objective evaluators.

Now let’s talk about the public. We evaluators are passionate about making this world a better place, striving toward an equitable and sustainable future. However, people outside our conference hotels do not know what evaluation is or what evaluators do. Teachers, accountants, physicians, engineers, social workers do not need to explain to anyone what their professions are all about – but evaluators do. This is because we evaluators primarily talk among ourselves and do not interact much with others outside of our profession.

At this year’s closing plenary in Denver, Past AEA President Rodney Hopson asked whether we make a positive footprint in cities where we hold annual conferences. Does anyone in Denver know that we are here? Did Washington, DC know that we were there last year? Does anyone in Chicago know that AEA is coming to their community in 2015? In essence, he challenged us to think about the larger purpose of our profession and the association.

The fourth P is the press. More than 3,000 evaluators gathered in downtown Denver, but do we know if the local press covered our conference? Did they publish any story about us? Of the hundreds of conference sessions held, did we have something worth sharing with people outside the hotel? If yes, then we should have reached out to the press because the press could be a link between us and the public.

Unless we discuss these 4 Ps in greater depths at our conferences, in our journals, and in our classrooms, I do not believe we can have the kind of impact we would like to see through our evaluations in moving toward an equitable, sustainable world.  Evaluation is inherently linked to the 4 Ps. We must demonstrate the relevance of evaluation to policymakers, the public, and the press. Evaluators do not have the authority to make public policies; policymakers do. Therefore, we have no choice but to engage policymakers in evaluation activities. We should reach out to the press, which can carry our evaluation message to the public – policymakers tend to pay attention to their constituents.

Riitta, you too have emphasized the importance of working with political decision makers, and more broadly with colleagues who are responsible for evaluation demand and use, including commissioners of evaluation? Why do you think this is not automatically happening?

Riitta: One reason may be the curse of the principle of independence. When independence starts dominating the evaluation function, it very easily leads to self-centred isolation and irrelevance. Too often we forget that evaluation is guided not by a single but a set of principles. It is important to complement independence with relevance and responsiveness. Evaluation is a sport where you can only win a gold medal if you master this triathlon. J

I also think that we need to learn how to work with important partners beyond our own community. A part of this is to communicate in a non-jargon language. Rakesh, you have been successful in working with politicians. What lessons can you share on how best to approach non-evaluators?

Rakesh: Riitta, you are right about the curse of the principle of independence. Many evaluators are concerned that they will lose their independence, and hence their credibility, if they engage with policymakers. However, to conduct evaluations that are used by policymakers, we have to be relevant and responsive to their information needs.

Responsiveness involves understanding the political context of the evaluation; identifying key stakeholders and knowing competing, conflicting, and complementing interests of those stakeholders; and establishing relationships and building trust with key policy actors (includes policymakers and those who have access to policymakers, and those who influence the policymaking process). I do not believe we have a choice between independence and responsiveness; we have to do our best to maximize both and to know when and where to draw the line when independence is threatened so as not to compromise our professional standards and ethics.[2]

Riitta: Agree. I think it is important for evaluators to remember that they are only evaluators. There are situations where evaluators, for example, working in the area of planning come very close to assuming the role of the planners. Evaluators should remember their role as visitors in processes that are owned and hosted by other professionals. Evaluation is a service function to these colleague professionals. Evaluators cannot and do not need to master everything. Partnerships are all about trust among professionals and reciprocal learning.

What would it take to be successful in this beyond-evaluation approach?

Rakesh: To be successful, I believe we will need to step outside our comfort zone and reach out to those who make and influence public policies that affect our lives and our environment. It requires building new partnerships with non-evaluation organizations that have access to policymakers and understand the complexities of public policy environments. These partnerships must be forged without sacrificing important principles of evaluation, such as independence, non-partisanship, and transparency in the public interest.

Riitta: How do you see the role of VOPEs or the Voluntary Organisations for Professional Evaluation (i.e., evaluation societies and associations) in building new partnerships?

Rakesh: Evaluation societies and associations across the globe are the perfect vehicle to promote evaluation use among policymakers. Individual evaluators, evaluation offices, and evaluation consulting firms do not have the needed capacity to reach out to policymakers and the public. However, VOPEs working together can. For example, Asian and African evaluation associations in collaboration with the EvalPartners are now making efforts to engage policymakers. The involvement of parliamentarians at the European Evaluation Society’s biennial conference in Dublin is an excellent example of what needs to be done worldwide on a consistent basis.

Thank you much, Riitta. I’m so glad we met in Denver. It was a pleasure to discuss with you this important topic of building new evaluation partnerships. That, in my view, is the evaluation future.


[1] Policymakers refer to elected public officials like parliamentarians, members of the US Congress, provincial ministers, and state legislators.

[2] Evaluator Advocacy: It Is All in a Day’s Work. Rakesh Mohan, American Journal of Evaluation, vol. 35, September 2014, 397-403.

Add comment

Log in or register to post comments

Comments

Riitta and Rakesh, thank you for interesting blog post/discussion. I agree with the points you made and advocate for more open approach and close communication/coordination/cooperation with all 4Ps. Somehow, I believe (not evidence based) that most of the evaluators would agree with you, but also that many of us just overlook the importance of 4Ps. I am especially interested in the role of the press and other media in evaluation. From my perspective they could be good ally not only in reporting on some evaluation event, but also, and even more in promoting evaluation culture; asking for evidence based policies and results; and finally in disseminating evaluation conclusions and recommendations (when appropriate). With this in mind I was thinking about creating a small training about evaluation for journalists. My idea was to introduce evaluation as concept to them, not to discuss about methods and approaches, but to present an evaluation as a way of thinking and valuing. My question for you and other colleagues is - do you know if someone already done something similar? Do you have any experience to share? Finally, what do you think about the idea? Thanks, Ivan @sociosolidarity @evaluacija www.evaluacija.com

Ivan, Thanks for your comments that expand the discussion to the area of press – top importance & top challenge at the same time! Your idea of training for press is an excellent one. Two things I wanted to comment on: First, your approach is a very good reflection of a comprehensive approach that is needed in evaluation capacity development. We often implement evaluation capacity development (ECD) by training evaluators, but that is far from adequate. A comprehensive approach addressing both evaluation supply and demand; addressing individuals, institutions and the operating environment (including press) is needed. Secondly, it would be interesting to bring together both evaluators and press for learning together. While the press needs to understand evaluation, evaluators need to understand press. I think this can be easily achieved by involving representatives of both professions in training - understanding the basics, analysing cases, simulations, developing communication plans ... and ... SO many opportunities for learning in partnership. I do not have info of previous evaluation training involving press. I’m very interested to learn from your experiences –please keep us informed!

Riitta, thanks for the response. Yes, you are absolutely right, it is a challenging issue. I am still in the planning phase, drafting it and exploring possibilities - more and less expensive ones :). Joining training is one of the options I was thinking about, too. From my point of view, and based on my experience, combination of training and real world practical assignment would be the best mixture. Ideally, it would include four steps: (1) few days of training; (2) month or two for home based practical assignment (that could be done in groups, and remotely supported by experienced evaluators); (3) follow up event for reflection and experience exchange between participants/groups; (4) finally, publishing evaluation stories in media. I know, this is expensive one, and in Serbia this can be done only if it is supported by donors. But, we will try. Other option is to do "only" a training. WIll see. Curriculum is also a challenging issue for me. What journalists have to know and what is good to know about evaluation? What will arm them sufficiently? I am thinking about starting it with general intro about public policies; then role of the citizens, media, CSOs in each phase of policy life-cycle; and after it to go in evaluation. As I said, still drafting it.

Dear Ivan: Thank you so much for your kind words and excellent comments. I agree with both you and Riitta. The credibility with the press my office and I enjoy today has taken us 12 years to build. Based on my experience in Idaho, I firmly believe that building any lasting relationships and partnerships is an incremental process that takes a long time and is totally dependent on mutual trust and respect. I love your idea of organizing training on this subject. I wish more evaluators will follow your lead. My suggestion would be two-fold: (1) make sure the trainers have practical experience in working with the press extensively for promoting the use of their evaluation work and educating the public through their evaluation findings, and (2) invite highly respected journalists (both men and women) to participate in discussions at your training. Both evaluators and journalists have much to learn from each other. Both professions are interested in telling the “truth to powers” and making our world a better place for all. I would love to hear about your efforts. Let me know if I can be of any assistance. Best of luck and Namaste.

Here is an outline of the policies and process for interacting with the press we use in our office: (1) No one is allowed to talk to the press unless I am present during those conversations. In our office, the evaluator who does the work gets to talk to the press. (2) We do not make any partisan comments, nor do we comment on any legislation. (3) When talking to the press, we limit our comments to the evaluation work that we have done. (4) We trust the reporters we talked with. (5) The reporters know in advance that we would not comment on the legislation being considered. (6) I trust my staff that they will follow office policies and will use sound judgment in responding to the reporters’ questions. (7) Even though we are only an office 9 people, we are an independent agency of the Idaho State Legislature. As the director of the independent agency, I am responsible for making final decisions about whether and when to talk to the press. In the end, I am accountable to the Idaho State Legislature and to the people of Idaho.

While I agree with the diagnostic - evaluation has had limited press access, no public profile, marginal influence on politics and a mixed record in engaging policy makers - I completely disagree with the remedy. Kowtowing to policy makers is the problem rather than the solution...Lack of professional autonomy, timidity towards power holders and fee dependence are the main curses that plague our discipline and keep it in the dark. Without independence evaluation is taken for granted and cannot compete with the management consultancy industry. With more independence we would speak truth to power, arouse controversy and contribute to democratic debates. The press and politicians would pay attention to us. We should seek to be respected rather than loved. Making use of evaluation tools is not evaluation. Evaluation has a higher calling: making the world a better place.

Dear Bob, many thanks for your inspiring comments. Let me first agree that independence is the most important and unique added value of evaluation. But independence alone does not make evaluation meaningful. Evaluation that does not have a service function is an investment wasted. Working together with users of evaluation results is not kowtowing. It is ensuring a return on investment. The idea that evaluators have a monopoly on truth (or making the world better!) has always been very alien to me. Many planners, policy makers and politicians are as committed in truth and a better world as evaluators are. Evaluators must have the strength to speak in situations of controversy. But not being loved cannot be an end in itself. What would be your advice in situations where there is no major controversy but the challenge is simply getting evaluation results used? Dear readers, I’m writing when snow is finally falling and painting Helsinki white. I wish you all a happy holiday season, and look forward to an exiting EvalYear 2015!

Thank you much for your response, Bob. I am encouraged by the fact that you “agree with the diagnostic.” With respect to your disagreement, I am not sure what part of the blog or our (mine and Riitta’s) comments gave you the impression that we are advocating for kowtowing to policymakers, lack of professional autonomy, and timidity toward power holders. Our blog talks about responsiveness AND independence, which is like doing a high wire act – it requires skills, judgment, and above all, courage. As Riitta said, evaluators are not the only truth seekers. Like evaluators, many policymakers, journalists, and others are also interested in making this world a better place – maybe their perspectives and approaches are different than yours and mine. In Idaho, I have had the privilege to work with some amazing people of different backgrounds (including those who did not agree with us) – they helped us translate our evaluation work into something meaningful and useful. We don’t have to seek neither respect nor love. Our credibility comes from the work that we do and the manner in which we carry out our responsibilities. Here are links to two press articles that will clearly show that responsiveness AND independence can go together and there is no need to kowtow to anyone. In the public policy arena, we all have to work together, learn from each other, and try to see the truth through the lenses of those who disagree with us. (1) https://idlso.box.com/s/5n7wzem07p4dzm4mdjv3 (2) http://www.spokesman.com/blogs/boise/2014/jul/28/senator-suggests-secret-review-releasing-agency-performance-evaluations/

Rakesh Thank you so much for the clarification and for the illuminating references.. Given my core belief that independence is central to the discipline and the obvious observation that vested interests often seek to capture our work I apparently over-reacted to (or misunderstood) Riitta's statement that evaluators are cursed by their independence.. I fully agree that we are not the only truth seekers and that independence is not isolation. But ours is indeed a high wire act and I am sure you and Riitta would readily agree that there are limits to responsiveness (e.g. that we should not be responsive to the wrong values) and also that promoting the public good rather than pleasing our clients should be our primary concern.. Michael Scriven's EES keynote presentation said it all: ethics is the last frontier. In Riitta's useful triathlon I would privilege independence since it is currently undervalued by the market based, fee dependent approach to evaluation commissioning. Best Bob

Bob, thank you much for explaining your point of views. I totally agree with your following comments: (1) “independence is central to the discipline,” (2) “there are limits to responsiveness,” (3) “promoting the public good…should be our primary concern,” and (4) “ethics is the last frontier.” I’m sure you know that practicing these values in day-to-day evaluation work is nothing less than a high wire act. It is challenging, exciting, rewarding, and at times frustrating. In short, it is damn difficult. This high wire act also carries a great deal of risk for evaluators and evaluation organizations – their credibility could be tarnished and livelihood could be destroyed by detractors. Bob, I hope you will join me and Riitta in broadening the scope of evaluation discussions to include the 4 missing Ps. I have much to learn from you and others on how to best accomplish this goal. Again, thank you and Happy Holidays.

Thank you so much Riitta, Rakesh and Bob for quite honest and frank deliberations. Your views in general are quite on a trajectory that will be highly beneficial to evaluation practice globally. I must still go back first to reinforce Bobs view on ethics being the frontier. However, evaluators cursed themselves due some of the limitations mentioned by Bob. I think a sound understanding of how to use the politicians is central and very important to evaluation. While we are looking at independence we should also be looking at the relevance of interdependence and interconnectivity of our work, AEA President Beverly Persons illuminated this clearly in the plenary session in Denver. The transitory nature of evaluation makes it critical to have the skill to interpret the political economy of the evaluation we undertake and understand the interdependence and connectivity, while maintaining sound independence. This will have a huge potential on the use of our work while we are far gone. We also take with us a gift about an additional experience on relating with a complex group. About the same period just across the road at the Hilton in Denver at the African Evaluation Association Reception, I was making a presentation about the need to bridge the gap between Evaluation and politician/policy makers, "The New Face of AfrEA". They are in need of our help and we are most in need on their understanding. AfrEA, EES and Evalpartners and indeed many VOPEs are now taking the big bold step to face our perceived demons. Rakesh the 4Ps are quite critical and based on local experience, I believe given your position you will understand better than I do that if we rally get right how we engage and use the P3 (Public) and P4 (Press) to influence evaluation, we can achieve a lot on the other 2ps (policy makers & politicians) with less prompting. Greater constructive and strategic engagement of the press and the citizens can build the required mass and positive pressure to influence change and provide voice to evaluation. The latter two have little or no threatening potentials to Evaluation Independence rather they are ethically partners in promoting accountability (Use) and transparency of evaluation results. I think we have two things to change internally as evaluators first maintaining ethical frontiers (diplomatically encapsulated by Bob) and building our skills in political economy and advocacy to better position and maintain our independence. Rakesh it will be nice to learn how your work is locally influencing decision making at the Idaho legislature. Riitta is a shining example too of what we can achieve with politically smart evaluators towards a sustainable future but also a good service provision that meets the needs of 3I's Independence, Interdependence and Interconnectivity. As Riitta said at the AEA closing plenary " I think it is now time for Evaluators to Open their eyes". Once again thank you for pushing to do the first brief comment on Evaluation in 2015.Happy New Year and Best wishes to Riitta, Rakesh, Ivan and Bob. It is an honor.

Thank you for your thoughtful comments, Suleiman. I will be happy to chat with you through email about how my office works with policymakers and the success and challenges we have had working with policymakers. To continue the conversation off line, just send an email to me at [email protected] I really like your 3Is (independence, interdependence, and interconnectivity). I think you and Riitta should write a guest blog about it. The concept of 3Is complements my 4Ps. Again, thank you and best wishes for 2015.

Dear Adam, Many thanks for your comments and a happy EvalYear 2015! Africa is setting an interesting example on partnership with parliamentarians. We will all be following and eager to learn from the APNODE (African Parliamentarians Network on Development Evaluation) initiative. I’ve been impressed by the commitment of APNODE members and the swift progress in getting the network operational.

The following comment was made in an off-line email to me by Graham Shaw. I thought EES members will benefit from Graham's insight on the issue of four missing Ps. I asked for his permission to post his comment here. He kindly agreed to my request. Here is his comment: Thanks to you Rakesh for starting this discussion and for all those contributing to the thread. I will try to explain the thinking behind my comment about constraints due to loyalties or positioning. I believe that in the areas of politics and policymakers, the importance of perceived utility is highly reliant on values (thinking here of the important 1980 paper by Carol Weiss and Michael Bucuvalas). It is increasingly clear that findings or knowledge alone are seldom influential without relationships or trust and these appear to rely (in many if not most cases) on perceptions of being on the same side in important ways (often called coherence or alignment). Jonathan Haidt's "The Righteous Mind" shows that individuals vary considerably on their unconscious "moral foundations" and these foundations appear to influence values. Specifically, he shows that some people place the moral imperatives to look after one's own, or to uphold the structures of a society, above the imperative to give all people an equal chance. Similarly, Eleanor Chelimsky in her recent article on public-interest values points out controversy on fundamental national values and - for example-in specifying that "citizens" should have equal opportunities, there is an implication that citizenship is an important aspect of worth - much as in-group status can be seen as relevant to worth in other circumstances. These views contradict the appealing idea put forth by Michael Scriven in his EES keynote on ethics that the foundational idea of equal rights is uncontestable. To the extent that those close to us are seen as having greater value than others, or that we must sacrifice the rights of some to achieve the rights of others, loyalty and position seem to matter. In a more general sense, the social construction of knowledge and what is perceived as truth and of value cannot help but influence what is done (if not what is possible) in the press and attempts to influence or persuade the public. (I am thinking of the research into diffusion of innovations by Everett Rogers and colleagues). Given the limited influence of evidence that is presented outside of relationships and trust in shared values, recognizing the place of values alignment in working with the four Ps seems both vital and constraining. I look forward to learning from the discussion. Graham Shaw