c2015 Projet de programme / Draft program

Transcription

c2015
Projet de programme / Draft program
MISE À JOUR
/ UPDATED 2015-04-08
TABLE DES MATIÈRES
DIMANCHE 24 MAI – INFORMATIONS DÉTAILLÉES
TABLE OF CONTENTS
SUNDAY, MAY 24 – DETAILS
LUNDI 25 MAI – INFORMATIONS DÉTAILLÉES
MONDAY, MAY 25 – DETAILS
MARDI 26 MAI – INFORMATIONS DÉTAILLÉES
TUESDAY, MAY 26 – DETAILS
MERCREDI 27 MAI – INFORMATIONS DÉTAILLÉES
WEDNESDAY, MAY 27 – DETAILS
24 MAI
DIMANCHE SUNDAY
MAY
24
13h-16h
Approche fondée sur les droits de la personne: Un regard sur l’évaluation axée sur le genre
Jeiran Rahmanian, Vanessa Anastopoulos
ATELIERS/WORKSHOPS
Consulting Challenges
Gail V. Barrington
ATELIERS/WORKSHOPS
Les protocoles d’évaluation faisant appel à des méthodes mixtes
Jacques Bérard
ATELIERS/WORKSHOPS
Participatory Evaluation: Learning for a Change
Linda E. Lee
ATELIERS/WORKSHOPS
Skills for Visual Data and Non-Lethal PowerPoint Presentations
John Burrett
ATELIERS/WORKSHOPS
9h-12h
Évaluer les réglementations pour mieux anticiper leurs effets : Balises pour fiabiliser les Analyses d'Impacts
Réglementaires (AIR)
Eva Anstett
ATELIERS/WORKSHOPS
Facilitation skills for evaluators
Jennifer Birch-Jones
ATELIERS/WORKSHOPS
Shared learning and participatory evaluation: the systematization approach for the assessment of
development interventions
Esteban Tapella, Pablo Rodriguez Bilella
ATELIERS/WORKSHOPS
Understanding and Using Contribution Analysis
Kaireen Chaytor, John Mayne
ATELIERS/WORKSHOPS
Using Microsoft Excel to Enhance Data Analysis in Evaluations
Sandra Sellick
ATELIERS/WORKSHOPS
2
9h-16h
Qualitative Work in Evaluation: Why? When? and How? For the World We Want
Cindy Tananis
ATELIERS/WORKSHOPS
3
25 MAI
LUNDI MONDAY
MAY
25
10h-10h30
Évaluation d'une formation en matière d'agression sexuelle envers les enfants autochtones
Xavier Barsalou Verge, Renée Séguin, Mélanie M. Gagnon, Christian Dagenais
PRÉSENTATION ÉTUDIANTE/STUDENT PRESENTATION
Le centre d'expertise Marie-Vincent offre un programme de formation en matière d'agression sexuelle envers les
enfants aux communautés autochtones du Québec. Tout au long de ce projet échelonné sur deux ans, une démarche
évaluative est menée en parallèle. Les objectif de cette démarche sont multiples, mais ont un but commun :
s'assurer que le programme de formation ait réellement une utilité pratique et qu'il réponde aux besoins de ces
communautés. L'exposé s'attardera à décrire la démarche évaluative qui comprend de s collectes de données
quantitatives, ainsi qu'une étude de cas multiple pour en apprendre plus sur l'utilité pratique de la formation en
contexte autochtone. Il sera aussi question des efforts mis en place pour retransmettre les résultats de l'évaluation
aux participants et à leur communauté. Pour conclure, les enjeux et les leçons tirées de cette évaluation menée
auprès de populations autochtones seront présentés.
10h30-11h15
Is Evaluation Losing Salience?
Andy Rowe
CONFÉRENCE D’EXPERT/EXPERT LECTURE
The connectivity of natural and human systems is demonstrated by important shifts incorporating sustainability into
public policy, the actions of for-profit organisations and public awareness. Climate change and sustainable
development are but two demonstrations of this change. To be salient today analysis and advice must address
sustainability. The premise of this presentation is that evaluation as a field is still strongly focused on the past when
a focus on human systems alone was OK. It is no longer OK, evaluation risks losing salience and falling even further
behind on our use agendas. For evaluation to regain salience it needs to incorporate sustainability into training,
standards, practice and theory, and the stance of evaluation to contemporary siloed go vernance. Suggestions are
offered and audience discussions of the issue and ways forward will be invited.
10h30-11h30
A Participatory Approach for Analysis and Reporting to Enhance Evaluation Knowledge
Shannon Clark Larkin, Nicole Michaud, Michelle Picard-Aitken
EXPOSÉ/PAPER
The use of a streamlined and stakeholder engagement approach for analysing and reporting on evaluation findings
and recommendations for the evaluation of the Canada Excellence Research Chairs Program will be presented. Given
tight timelines, and complex data collection, governance and consultation processes, an approach was implemented
for the analysis and reporting phases that enhanced stakeholder participation and resulted in increased access to
evaluation knowledge across the various stakeholder groups. Details on the preparation and usefulness of a
presentation deck of evaluation findings that was co-produced by the internal and external evaluation team will be
provided. The deck was refined and updated based on feedback from various stakeholder groups during the analysis
phase. This resulted in an evaluation report that contained concise and actionable recommendations that addressed
both accountability requirements and continuous improvement for program management.
Bringing the coaching attributes to evaluation
Kassem El Saddik
4
EXPOSÉ/PAPER
The paper intends to bring the co-active coaching attributes and perspectives into the evaluation practice. It will
highlight key similarity between the two practice in terms of principles, ethics, qualities and strategy. It argues that
applying the fundamental elements of co-active coaching in evaluation will take the latter to an extra level by
unveiling new dimensions, revealing innovative theory of change and uncovering unintended outcomes. It sh eds the
light on the co-active coaching cornerstones (holding people naturally creative and resourceful and whole),
attributes (self-management, listening, curiosity and deepening the learning), setting the strategy (designing
alliance) as well as the ethical and paradigm (client-centric, unveiling intended and unintended outcomes).
Developmental Evaluation Learnings from a Girls Empowerment Program
Anne Miller
EXPOSÉ/PAPER
Sirius Girls is a pilot program in partnership with United Way's All in for Girls In itiative and the Women's Leadership
Council. It is an after school program for vulnerable girls in grade ten using a social -emotional and service learning
approach to assist girls to take action with the support of adult allies (women Leaders who volunteer with the
program) to experience mastery and build confidence to take new risks, creating a better community for all girls to
thrive. Since the program is an innovative initiative at the development and pilot stage, a developmental evaluation
approach has been used to help understand the successes, challenges and positive impacts of the program. This
presentation will describe the developmental evaluation approach taken with this program, new measurement tools
(like the VACO survey) that have been explored for use with this population, and learning from the process.
Évaluation des effets de la pratique de l'évaluation d'impact sur la santé (ÉIS)
Jean-Marie Buregeya
L'évaluation d'impact sur la santé (ÉIS) est une démarche évaluative utilisée afin d'analyser les effets sur la santé des
politiques publiques qui ne relèvent pas du secteur de la santé (Harris et Spickett, 2011). J'applique une approche
novatrice (Analyse de contribution) pour l'analyse des effets à un programme complexe, en particulier pour le projet
de revitalisation du Vieux-Sorel et le projet d'urbanisme du centre-ville de Chateauguay afin d'apprécier le potentiel
d'ÉIS pour bonifier cette politique et ses impacts sur la santé et l'équité. Dès lors, il est nécessaire de présenter le
protocole de recherche puisque l'analyse de contribution est indiquée aux cas où les méthodes statistiques, tels que
les essais randomisés et quasi-expérimentaux, ne sont pas les mieux indiquées pour établir la causalité d 'une
intervention aux effets observés (Mayne, 2012), et n'a pas été utilisée pour évaluer les ÉIS.
Evaluation for the world we want: Child Rights Impact Assessment (CRIA) for Business
Tara Collins
EXPOSÉ/PAPER
'Evaluation for the world we want' should include attention to human rights and business in order to support a
better world. The international discourse has done so as evidenced by the United Nations Guiding Principles on
Business and Human Rights (2011) and the Children's Rights and Business Principl es (UNICEF, Save the Children, &
Global Compact (2012)). These documents identify impact assessments to evaluate effects upon human rights. But
how can the child rights impact of business be assessed in practice? The implications of this relationship must be
ascertained through the development and use of child rights impact assessment (CRIA) for business. However, this
activity is emerging with restricted participation. As there are significant challenges to identify and assess processes
and results, other actors including evaluators are needed to support progress. This paper presentation presents a
preliminary CRIA tool to contribute to the evaluation field.
Evidence-Based Principles to Guide Collaborative Approaches to Evaluation
J. Bradley Cousins
5
This expert lecture will focus on practice in collaborative approaches to evaluation with particular emphasis on the
development, validation and ongoing international field testing of evidence -based principles. Featured will be the
work of a collaborative research team headed by Cousins, Whitmore and Shulha that has been unfolding over a 3
year period (see Cousins, Whitmore & Shulha, 2013, 2014; Shulha, Whitmore, Cousins, Al Hudib & Gilbert, 2015).
Specifically, the lecture will: (i) describe a bottom-up, four-phase empirical process to develop and validate
principles involving 320 evaluators from CES, AEA and IDEAS; (ii) introduce the set of eight (8) validated principles
and (iii) report on ongoing international interests in field testing the principles. A goal of the lecture is to engage
with audience members in the identification and mobilization of field testing opportunities.
How to publish in a journal of evaluation?
Robert Schwartz, Kristelle Alunni-Menichini, Lynda Benhadj, Jean-Marie Buregeya, Marie Beausejour
Dissemination of practice notes, real-life cases addressing challenges, and research and program evaluation findings
in evaluation’s domain have a primordial importance in order to change professional practices and influence
decision making in public organizations. Since the publishing process in the field of evaluation can be complex, Dr.
Robert Schwartz, Editor-in-Chief for the Canadian Journal of Program Evaluation, will be presen ting the specific
characteristics of his journal and guidelines on 'what to do or not to do to publish in evaluation' and this, in
collaboration with EASY research chair’s members. He will also answer frequently asked questions addressing the
publishing process in evaluation. We invite you to participate at the preparation of these questions by sending yours
at [email protected] before April, 1st. This expert lecture is thus an excellent opportunity to learn about
important tips to support publication of evaluation work.
Increasing the Evaluator's influence: techniques and micro-practices
Natalie De Sole, Melanie Hwalek
TABLES RONDES/ROUNDTABLES
Evaluation's influence is conditioned by the stakeholders' receptiveness to listen, make sense, and use f indings. The
door of opportunity often closes after evaluators turn in the final report and meet one last time with our client. The
nonprofit's program director, foundation's program officer, or the governmental policy advisor must champion our
finding's use or file them away. A plethora of models exist on how to engage stakeholders yet discussion is limited
on the nuanced techniques required to bring these models to life. At this roundtable participants will be shown good
practice models related to rules of engaging individual, group and organizational stakeholders to create change.
Participants will be asked to share insights about how they apply engagement models in specific contexts. Together
we will identify techniques and micro practices that increase stakeholder buy-in and the eventual adoption of
evaluation findings.
L'approche participative en évaluation : le cas de Pouvoir Partager/ Pouvoirs Partagés
Joanne Otis, Lyne Massie
EXPOSÉ/PAPER
Pouvoir Partager/ Pouvoirs Partagés (PP/PP) est une intervention conçue pour outiller les femmes vivant avec le VIH
(FVVIH) face à la question du dévoilement du statut séropositif au VIH. S'appuyant sur un devis mixte, PP/PP a été
évalué au Québec et adapté culturellement au Mali (Gundo So) grâce à plusieur s recherches participatives où la
collaboration entre les différents acteurs et l'implication des FVVIH sont essentiels. Réfléchi par une équipe de
chercheuses AVEC et POUR les FVVIH elles-mêmes, PP/PP contribue au renforcement d'un plus grand pouvoir d'ag ir
chez les femmes du Québec et du Mali relativement au dévoilement de son statut séropositif au VIH. Diverses
stratégies de partage des connaissances et de renforcement des capacités ont permis la coproduction d'outils de
pratiques professionnelles disponibles en ligne (www.pouvoirpartager.uqam.ca), contribuant ainsi à améliorer la
qualité de vie d’'un plus grand nombre de femmes.
6
L'évaluation pour optimiser le dépistage du VIH pour les hommes gais: le projet SPOT à Montréal
Ludivine Veillette-Bourbeau, Joanne Otis
EXPOSÉ/PAPER
SPOT est une recherche-intervention offrant, depuis 2009, un counseling et un dépistage du VIH rapide, anonyme et
gratuit en site communautaire pour les hommes gais ou bisexuels. En soutien à l'évaluation des effets, une analyse
d'implantation a été menée. Un monitorage du degré d'implantation a permis de voir sa fluctuation et des analyses
de régression linéaires d'identifier les éléments associés à sa variation, le counseling étant adapté au participant. Un
processus d'implantation en plusieurs phases, modulé par des facteurs liés aux motivations des acteurs, à la
complexité des dynamiques partenariales, aux défis de la coordination et à l'organisation de l'équipe terrain a
émergé de l'analyse qualitative. La pérennisation de SPOT soulève des défis: transformation d'une intervention en
contexte de recherche vers un service régulier, maintien des acquis de la recherche et appropriation de la gestion
par les partenaires.
Managing project constraints for the next decade
Judy Lifshitz
Evaluations are a key source of information to assist in responding to specific questions to support decision making
in a time where resources to conduct them are more and more limited. Thus, it is crucial that evaluatio ns be on
time, on budget and in scope so that the evaluations can be useful for decision making. This presentation will
provide an overview of the international standards used in the field of project management to help to address the
three project constraints of time, budget and scope. Examples and practical tools and techniques for addressing
time, budget and scope will be shared with participants based on the presenter's experience and internationally
recognized best practices. As a PMP, over the last five years, Judy has managed several evaluations and has used
these tools and techniques to ensure that timelines, budgets and scope is managed. She has done previous
presentations on project management at CES and PMI related events.
Reframe QUTs evaluation framework encompasses a multi-faceted approach to meet stakeholder needs
Lyn Alderman
Since 1984, the Australian government has invited students to provide feedback on their course of study and in the
early 2000s annual student feedback surveys became a government accreditation requirement for universities. As a
result, Queensland University of Technology (QUT) launched online invitational student feedback surveys in 2007. In
2010, stakeholders raised concerns about the mixed purpose of the surveys, the potential for data misuse and how
academics felt evaluation was being done 'to them' rather than 'for them'. In response, QUT introduced Reframe a
five-year project encompassing three phases: stakeholder engagement and discov ery - executive support,
environmental scan, literature review, and theoretical concept; product development; and dissemination and
communication - target audiences, communication plans, professional development and iterative refinement. In
2015, QUT's broad evaluation approach meets the needs of a range of stakeholders.
The experience of a paticipative evaluation: Brazilian QualiSUS-Network Project
Gisela Cardoso, Marly Marques Da Cruz, Aline Duque De Macedo, Ana Cristina Gonçalves Vaz Dos Reis,
Celita Almeida Rosario, Patrícia Pássaro Da Silva Toledo, Solange Kanso
EXPOSÉ/PAPER
The Brazilian QualiSUS Network Project (QNP), a cooperation between the World Bank and the MoH, is a strategy to
promote the qualification of health care assistance and management, and the development of technologies among
Health Care Networks. This intervention was implemented in 15 different and diverse health regions. In order to
support the institutionalization process and use of M&A as a tool for management and decision -making in context,
an implementation evaluation was conducted. The evaluation adopted a participatory approach which included the
development of 15 regional logical models, the identification of potential evaluation users, an agreement on
7
indicators and goals; the characterization of the political-organizational contexts; the identification of facilitators
and barriers in the implementation process, and the analysis of the agreed management arrangements and its
contribution to the implementation in the contemplated regions.
The UN Resolution and International Evaluation Year - What's in it for Africa?
Ian Hopwood
EXPOSÉ/PAPER
What explains the upsurge of enthusiasm for evaluation (UN Resolution, International Evaluation Year, etc.)? Will it
last? Can it bring real change? What are the driving forces? The author uses his wide international and African
experience to assess real potential and identify pitfalls. Can evaluation escape the “donor driven image” and
monitoring overhang? Are there pre-conditions for evaluation to thrive and make a difference in Africa? How does
evaluation relate to cultural context and governance systems? The links to policy and planning? And to public
pressure for performance and accountability? Correcting the imbalance between accountability and learning?
Shifting the focus from projects to systems and policies? Getting Africa’s leadership and donors to acknowledge
'failure' and embrace learning approaches? The conclusions highlight new partnership opportunities and capacity
development initiatives to foster African leadership for appropriate practice.
Valuation: Methods for Representing the Value of Program Impacts in Monetary Units
John Gargani
EXPOSÉ/PAPER
Increasingly, evaluators are being asked to perform valuations — measuring the merit, worth, or significance of
program impacts in monetary units. In particular, valuation forms an integral part of recent value -for-money policies,
social return on investment analyses, and pay-for-success initiatives. I provide an overview of valuation, describe
valuation methods currently used by evaluators, and demonstrate how and why these methods sometimes produce
contradictory results. I conclude by describing a system for reporting valuations that may better communicate their
relative trustworthiness, thereby supporting the many uses to which they are put by funders, private investors, and
policy makers. Throughout, I provide examples using data on prisons provided by the UK government. The data and
worked examples will be made available electronically.
Valuing evaluation power and the power of evaluation in “Speaking Truth to Power”
Sandiran (Sandi) Premakanthan
The main focus of this session is about valuing evaluation power and the power of evaluation in speaking evaluation
truth to power, those who create the demand for it, legislators, the public, heads of government departments and
agencies, program management, non-government organizations and donors. I have defined the terms evaluation
power and the power of evaluation and identified several sources of institutionalized evaluation power. They
include: governments through legislations, authority instruments, and policies, philanthropic foundations, financial
institutions, government aid agencies, United Nations (UN) agencies, non-governmental organizations (NGOs) and
numerous networks: evaluation societies, associations and international networks. The evaluation power of the
government of Canada, termed the “value model” and the creation of the power of evaluation and its use for
informed decision making is discussed.
What Environmental Assessment can Learn from Evaluation
Paul Kishchuk
EXPOSÉ/PAPER
The Yukon Environmental and Socio-economic Effects Assessment Act allows for consideration of social and
economic effects in parallel with environmental effects when resource development projects are scrutinized. Yet,
almost a decade after implementation of the Act, full and proper consideration of social and economic effects still
struggles to gain traction in Yukon's environmental assessment process. Drawing on his experience in the fields of
8
both socio-economic effects assessment and program evaluation, Paul will put forward a hypothesis, using the logic
model paradigm, which illustrates why socio-economic effects assessment has not been embraced in small Yukon
communities. The presentation will go on to explore how an evaluative approach to socio -economic effects
assessment can help advance resource development projects in rural and remote communities in Canada.
Whether Evaluations made a difference in the Management of HIV/AIDS Programs in South Asia?
R.S. Goyal
EXPOSÉ/PAPER
In the quest to seek desired impact, cost effectiveness and accountability, funders of HIV/AIDS programs have been
seeking systematic assessment of outcomes and impact of these programs. These evaluations were also expected to
become prime mover for continuation of resource allocation and scaling up of the interventions. This paper presents
a synthesis of evaluations of HIV/AIDS programs in South Asia particularly contributing to the program management
in relation to; communication, stigma and discrimination and, preventing HIV among adolescents and young people.
The broad inclusion criterions were; randomized control/ quasi-experimental/descriptive deign and evidence for
casualty/ what has or has not worked/ provide evidence for up- scaling/replication of interventions. Preliminary
analysis indicates that; curriculum based-teacher led interventions in schools; and youth friendly health services
have made appreciable contribution to HIV/AIDS programming in the region.
Working for a World We All Want: The Journey of a Community Needs Assessment
Denise Belanger, Linda Lee
EXPOSÉ/PAPER
While many practitioners embrace participatory evaluation, they are often faced with questions regarding how to
meaningfully involve participants in processes. What does meaningful participation and consultation look like in
different situations and with different constituents? Drawing from their work over the past few ye ars, Linda and
Denise will discuss collecting the voice of those who live in inner city neighbourhoods. In particular, the presentation
will focus on a community needs assessment undertaken in partnership with a community based agency where the
assessment focused on meaningful involvement of community, in the hope that those involved felt valued and
heard. In addition, capacity building was important to this project, as Denise and Linda worked with staff to
undertake community-based data collection and involve them finding meaning in the information collected.
Leçons apprises de l'évaluation du projet allo info sida
Rosine Flore Chuedo Djoungou
EXPOSÉ/PAPER
La stigmatisation liée au SIDA constitue un problème central pour les porteurs de VIH au Cameroun, car les malades
marginalisés tombent en déprime morale, ce qui complique leur suivi. Le gouvernement tente de résoudre ce
problème, dans le cadre de sa stratégie nationale de lutte contre le SIDA. C’est dans cette perspective que SUNAIDS,
en partenariat avec le CNLS a mis en œuvre le projet Allo Info Sida, dont l’objectif est d’apporter l’information, le
soutien psychologique et moral aux malades du Sida et aux PVVS. Il s’agit d’un dispositif d’appel téléphonique
gratuit accessible à partir du téléphone fixe et du téléphone portable de toutes les régions du Cameroun. Après 2
ans de mise en œuvre du projet, l’évaluation du projet a permis de mesurer l’atteinte des objectifs, d’analyser la
démarche de mise en œuvre du projet , dans l’optique d’envisager des persp ectives d’évolution et les possibilités de
repositionnement du projet face à l’évolution du contexte environnemental.
10h30-11h45
Web-Based Stakeholder Feedback Portals: Allowing Wider Stakeholder Feedback
Makenzie McPherson
EXPOSÉ/PAPER
9
Advances in technology provide evaluators with continuously improving, cost-effective methods for the
dissemination of results and opportunities for capturing stakeholder feedback. One of these mediums, a web -based
stakeholder feedback portal, provides evaluators the opportunity to post visual representations (e.g. process flow
maps, logic models, root cause analysis diagrams, etc.) allowing stakeholders to review, edit, provide feedback, or
validation. This presentation will provide a demonstration of an active web -based stakeholder feedback portal,
showing how it can effectively strengthen the evaluation, and how it might be adapted for other uses.
13h15-14h45
An innovative approach to link evidence & practice for practitioners in the employment services area
Susanna Gurr
EXPOSÉ/PAPER
The BC Centre for Employment Excellence aims to improve employment outcomes for job seekers by promoting
innovation and sharing evidence about employment programs with practitioners to enhance knowledge and ensure
best practices are implemented. The Centre has 2 key functions: 1) conduct projects in collaboration with
stakeholders to test innovative approaches 2)share research with practitioners. The Centre was launched in 2012 at
the initiative of the BC Gov't. The Centre emphasizes the role of practitioners in its activities to ensure the
knowledge developed and shared is relevant. Hear from the Managing Director about methods used to connect BC
practitioners to different forms of evidence in various engaging ways and the collaborations a nd projects that have
emerged from the Centre's activities. What are some opportunities to close the gap between evaluation and
knowledge capacity to enhance practice and ultimately lives of job seekers? What are some pitfalls to avoid?
Are user fees exemptions enough to increase the use of healthcare services by the worst off?
Nicole Atchessi, Valéry Ridde, Maria-Victoria Zunzunégui
User fees exemption programs have been implemented in certain low- and middle-income countries to improve the
populations' access to healthcare services. Given the considerable healthcare needs of the worst off, they should,
when exempted from user fees, be using healthcare to a greater extent than others. The aim of this study was to
assess whether user fees exemptions increased healthcare services use among indigents in the Ouargaye district in
Burkina Faso. The indigents' increased healthcare services use was not attributable to user fees exemptions. Some
contamination of the intervention is conceivable. Interventions combining user fees exemptions with actions
targeting other obstacles to healthcare access would probably be more effective in increasing indigents' use of
healthcare centres.
Collaborative evaluation capacity building for social innovation in the public sector through DE
Keiko Kuji-Shikatani, Scott Urquhart, Jennifer McMaster, Kieran McMonagle, Cristina Ilas, Angelika Kerr
PANEL/PANEL
Program evaluation began with the desire to seek information that can be utilize d to improve the human condition.
The panel explores how infusing evaluative thinking through DE is positioned as a responsibility for internal
evaluators and leaders at all levels of the system focused on organizational learning and improvement in its
relentless pursuit of reaching every student, including: collaboratively building relationships and holistically
nurturing the whole person as Four Directions circle of caring adults – school staff, parents, community partners, and
the ministry - so as to very purposefully position the sharing of caring of First Nations Métis and Inuit learners; and
partnering with three school boards to conduct case studies of existing teen parent programs that integrate
academic, life skills and counselling components with additional community services to support teen parents in
finishing high school and transitioning to further education, training or to the workplace.
Demystifying Gender Mainstreaming
Emmanuel Trépanier, Laurentine Mefire
10
EXPOSÉ/PAPER
Mainstreaming gender equality in international organizations is a challenging process that requires awareness,
political will, behavioural change, resources, time and, quite often, intensive guidance from specialized consultants,
using complex tools. Universalia's newly developed Gender-Sensitive Institutional and Organizational Performance
Assessment (GSIOA) Model is a user-friendly, time-efficient and free tool intended for users from all walks of life.
Adapted from the Universalia IOA Model, the tool allows users to assess the capacity, performance, external
environment and motivation of international organizations from a gender perspective in order to help decision makers improve equity in the workplace and contribute to positive social impacts. This presentation will allow
audience members to experience the GSIOA Model with pragmatic and challenging examples and increase their
acumen for equity.
Evaluating the Humanitarian Projects during Crisis: The Syrian Case
Tarek Daoud
IGNITE (PRÉSENTATION RAPIDE)/IGNITE-STYLE RAPID PRESENTATION
The prolonged crisis in Syria leads to catastrophic humanitarian conditions. According to UNOCHA (2014) report
300000 people are living in under siege. As a response, INGO’s launched a series of humanitarian projects in
coordination with local partners. Humanitarian organizations face challenges in evaluating the impact of the
implemented projects due to the problems involved in overcoming the constraints of physical access and the
obscure limitation of sharing evaluation results due to security purposes. I n this chaos situation, the participatory
evaluation approach found to be the best solution in order to provide an in -depth insight evaluation. Applying the
participatory evaluation improved projects performance, empowered the implementing partners evaluat ion skills,
creates evaluation teams in siege areas, enhanced the organizational learning growth, increased the validity of the
data collected, and strengthen the relation between local staff and external experts.
Evaluation du taux de survie à 12 mois des patients infectés par le VIH au Sénégal
Maguatte Ndoye Ndiaye
Contexte: Le taux de survie constitue un indicateur clé du programme et permet d'évaluer l'impact de la mise sous
ARV des personnes éligibles au traitement. Objectif: Évaluer le taux de survie des patients 12 mois après la DUS mise
sous ARV pour l'année 2012 Méthodologie : la formule de calcul est = (Cohorte actuelle (CA)/cohorte nette (CN) x
100 CN = Cohorte de départ (N) Transfert In (TI) — Transfert Ailleurs (TA) CA = CN — DC — PDV — ABAN — ARR
Résultat: le taux de survie nationale est estimé en 71% en 2012 et 73% en 2011. Conclusion: Disparité observée dans
les régions.
Evaluation for the World We Want? Emerging Lessons from the Global South
Robert Mclean, Mallika Samaranayake, Ziad Moussa, Colleen Duggan
PANEL/PANEL
In the view of many, evaluation can be used to help build the world we want. Canada's IDRC has supported this
vision through multiple avenues including supporting evaluation field -building in regions with identifiably less
capacity to match evaluation demand with local expertise. Several compelling results have emerged, and at
considerable scale. Two specific efforts have been the Community of Evaluators, South Asia (CoE) and the Middle
East and North Africa Evaluators Network (EvalMENA). Current President of the IOCE, Ziad Moussa will reflect on
experiences jointly leading the EvalMENA network. Founding President of the CoE Mallika Samaranayake will
comment on her work enhancing evaluation across South Asia. Insights will be of interest to those wanting to learn
unique lessons about professionalization, capacity-building and networking in evaluation. Especially of interest to
those working to bridge evaluation supply and demand between the global North and South.
Evaluation of Data Quality
Harold Henson
11
EXPOSÉ/PAPER
Problems with data have been an ongoing challenge in the field of program evaluation. In 2009, the Auditor General
reported that a significant share of published evaluations were compromised by problems with the available
performance measurement data. There are few reasons to believe that the situation has improved substantially
despite the improvements in technology. In this paper the author argues that the same basic process and
methodology used in program evaluation in general could be applied to the evaluation of data. It is argued that the
standardized evaluation questions and lines of evidence can be modified to assess quality of data generated by
programs for evaluation. At the end of this exercise there will be a report that resembles a program evaluation but
addresses the issue of data quality. In the best case scenario, high quality evaluations will be possible without
expensive surveys.
Evaluation Principles for a Democratic World
Elizabeth Lewis
As an organization with the mission to advance freedom and democracy worldwide, the evaluative efforts of the
International Republican Institute not only seek to adhere to internationally accepted standards for evaluations, but
to reinforce the democratic principles its programs seek to advance. IRI will use an example of a participatory needs
assessment it conducted in Somaliland to show how evaluation can consider six democratic principles: accessibility,
accountability, efficiency & effectiveness, equity & inclusiveness, responsiveness and transparency. Commissioned
by the UK's DFID, this needs assessment informed IRI's development of the Strategy for International
Democratization Support to Somaliland for the international community. The assessment captured input from
representatives of the international donor community, international implementers, subject matter experts and
Somaliland stakeholders from civil society, political parties, the media and government.
How to Use Process Flow Mapping for Continuous Quality Improvement
Ralph Renger, Melissa Rogan, Makenzie McPherson
EXPOSÉ/PAPER
Continuous quality improvement (CQI) is a common evaluation purpose. One potentially useful CQI method is
process flow mapping. Strategies for identifying and contacting potential subject matter experts, the basic steps and
tips in conducting a process flow map (PFM) interview, how to integrate interviews into a single, summary PFM,
methods for validating the PFM, and how efficiency data in conjunction with the PFM can be used to facilitate
corrective actions to improve system efficiency will be shared in the context of conducting CQI in a cardiac system of
care. Implications for using PFM in conducting CQI in other evaluations will be discussed.
How We Make a Difference
Paul Kishchuk
A mash-up of Mark Schacter's 'Keeping Busy...Making a Difference' logic model diagram and the DIKW (data information-knowledge-wisdom) paradigm is used to illustrate what the end game of evaluation sho uld be.
Influencing Change: Perspectives from Government, Private Consulting and the University
Donna Smith-Moncrieffe, Celine Pinsent, Mark Nafekh, Pierre Mercier
PANEL/PANEL
Influencing Change: Perspectives from the Government, Private Consultants and th e University A number of
evaluations conducted in Canada are produced by the federal government in collaboration with private consultants,
and educational institutions. It is important for evaluators from both the private and public sector to collaborative ly
reflect on how evaluation findings have had an impact on the communities and policies. This panel will answer
important questions about impact and influence from the federal government, university and evaluation
consultant’s perspective. The presentations will respond to some of the following questions as per the theme of the
12
conference: •How does evaluation make a difference in the community? •How has evaluation influenced policy in
the federal government and educational institutions? •What are some of the strategies used to reduce the barriers
to implementing sound evaluations?
Instrumentation de la théorie du programme dans le cadre d'une évaluation participative
Rodrigo Quiroz, Nathalie Bigras, Julie Dion, Karine Doudou
EXPOSÉ/PAPER
La théorie du programme (Chen, 1990, 2005) est une méthode utilisée par les évaluateurs afin de mieux définir
l'intervention à évaluer (Champage et al., 2011), de comprendre les processus à l'ouvre dans son implantation
(Green & MacAllister, 2002), d'identifier ses effets et les mécanismes en jeu dans leur production (Chagnon et al.,
2009), d'accroitre l'utilisation des résultats de l'évaluation (Wholey, 1987) et d'améliorer le programme (Weiss,
1995). Malgré son importance, il y a un manque d'écrits illustrant l'instrumenta tion de la production de la théorie du
programme dans le cadre d'une d'évaluation participative. Cet exposé vise à combler ce vide en présentant les
fondements conceptuels, ainsi que les étapes, les techniques utilisées et les principaux résultats obtenus de
l'évaluation d'un programme destiné à la formation des éducatrices en petite enfance.
L'analyse logique comme levier de changement : innovations et applications en santé publique
Tarik Benmarhnia, Véronique Foley, Jean-Marie Buregeya, Samantha Gontijo-Guerra, Gisela Cardoso,
Kristelle Alunni-Menichini, Marie-Claude Tremblay
PANEL/PANEL
L'utilisation de l'évaluation : ce que la théorie ne mentionne pas
Grégory Häuptli
Il existe une importante littérature sur l'utilisation de l'évaluation et plus particulièrement sur les facteurs internes
(qualité de l'évaluation) et externes (contexte politique et institutionnel) influençant négativement ou positivement
la probabilité que les résultats d'une évaluation soient utilisés par les chargés de programmes ou les décideurs
politiques. Cette présentation confronte cette littérature aux défis, contraintes et opportunités auxquels font face
les évaluateurs dans le but de mettre en avant les facteurs qui sembl ent être peu théorisés, voire ignorés par la
littérature. Il ne s'agit donc pas de remettre en cause la littérature sur l'utilisation de l'évaluation mais plutôt de
questionner pourquoi certaines contraintes vécues ne sont pas ou ne peuvent pas être théori sées.
Measuring the extent of institutionalization of ART programs in health facilities in Uganda
Henry Zakumumpa
In 2004,Uganda commissioned a national antiretroviral therapy (ART) scale -up program with external donor support.
We sought to measure the extent of institutionalization of ART programs in health facilities in Uganda and compare
institutionalization scores by health facility type. Level of Institutionalization (LoIn) scales developed by Goodman, et
al(1993)were used to measure the extent of institutionalization of ART programs at 195 health facilities in 42
districts of Uganda which received donor support between 2004 and 2009 to initiate ART services. The overall mean
institutionalization score for participating health facilities was 3.5(Range, 1-4) and the mean score for niche
saturation, the highest level of institutionalization, was 3.2(Range,1 -4).Private health facilities had the lowest mean
institutionalization score. Programs for enhancing the institutionalization of ART interventions in private health
facilities are recommended. ART program evaluation and supervision are deficient.
Not an Island - Insights from Other Disciplines
Alexandra Dagger, Shelley Borys, Bill Blois, Marie-Josée Dionne-Hebert
PANEL/PANEL
13
Evaluation has a long history of integrating knowledge and techniques from other disciplines. Approaching our work
with these other disciplines in mind brings fresh insight and adoption of new ideas and methods as well as benefits
to evaluands. This session will explore learning that evaluators can draw from areas such as performance
measurement, internal audit, risk and communications disciplines. Evaluators have not always tended to turn to
these areas for knowledge to use in our work but they have the po tential to add value. The panelists are senior
members of the federal evaluation community. However, although they come from the public sector, the disciplines
explored and lessons learned highlighted will be relevant to all evaluators.
Performance challenges of Tuberculosis control actions in Brazil: scapegoats, myths & possibilities
Gisela Cardoso, Elizabeth Moreira Dos Santos, Dolores Abreu, Pedro Paulo Chrispim, Gisele Oliveira, Juliana
Borenstein, Amanda Pereira
EXPOSÉ/PAPER
Tuberculosis (TB) is a national health problem for Brazil. The country has different cultural contexts and
epidemiological TB profiles. As a preliminary step of a performance evaluation regarding TB prevention and control
actions, the objective of this study was to conduct a strategic and logical analysis of TB control actions in four
Brazilian municipalities addressing activities at primary health care. To develop the strategic analysis, we focused on
a main question: "What are the potentialities of the national and municipal strategies to respond to TB control
challenges?". The steps of the strategic analysis included the identification of the strategic objectives, goals,
indicators and their classification according to the EGIPPS model domains and alignments. The appraisal of t he
logical modeling involved document review and interview with TB key actors. Results are going to be presented
according to the questions posed for both strategic and logical analysis.
Power, Institutions and Gender Relations: Can Evaluations Transform them?
Ranjani Krishnamurthy
EXPOSÉ/PAPER
This paper examines the perception of women, men, adolescent girls and boys from three Chennai slums, India how
the institutions of marriage, family, local markets and government should function if gender and social relations are
to be equitable. It then examines their perception of actual change in the last five years, what they consider positive
and negative and causality of change. In the process, marginalised people's inter -sectoral assessment of government
programmes also emerges (with recommendations) and this is contrasted with sectoral evaluations of government.
This paper argues that mainstream evaluations of government are sectoral or of one Department, and this does not
match the intersectoral and inter-institutional lived realities of women/girls and men/boys. Evaluations capturing
movement towards the world women/girls want, should give importance to their indicators of change & power,
methodologies, interpretations and recommendations
Program Design and Performance Measurement
Michel Laurendeau, John Burrett
EXPOSÉ/PAPER
While evaluation and performance measurement frameworks have helped, most evaluators still have to conduct
evaluations of public programs where the theory of intervention is uncertain and incomplete, with weak indicators
and poor ongoing performance measurement. Program research and descriptions of good practices often fail to
identify key success factors and support the improvement of program design. Frameworks have also been poor at
linking delivery and outcome indicators. The presenters will discuss two approaches to improved specification of
logic models with the use of: 1) network analysis to identify expected linkages of cause and effect and external
influences; 2) Integration Definition Function (IDEF) modeling techniques to describe delivery processes. They will
demonstrate how these techniques can support the development of improved logic models, a more compelling
analysis of program performance and a better understanding of program economy, efficiency and effectiveness.
14
Program science in practice: Lessons, challenges and implications
Sean Rourke, David Seekings
EXPOSÉ/PAPER
Program science is an emerging public health approach that systematically applies scientific knowledge and evidence
to improve the development, implementation and evaluation of programs. By emphasizing the importance of the
context in which interventions occur, program science highlights the essential role evaluation plays in informing
strategic action to address key health challenges. This research project will build a better understanding of what this
approach looks like in practice, using evidence from case studies of community led HIV prevention interventions to
assess the practicalities of making program science work on the ground. The presentation will examine the
challenges and opportunities of using evidence to inform each stage of community -based interventions, from
development to evaluation, highlighting key lessons from the cases. It concludes by discu ssing the broader
application of program science beyond public health to evaluation and public policy more generally.
The Influence of Internal Evaluation on Program Management
Andrea Macdonald
EXPOSÉ/PAPER
Nonprofits are under increased pressure to deliver more for less and to demonstrate results of their work. When the
Clean Foundation, an environmental nonprofit based in Nova Scotia, invested in evaluation training for management
and core program staff, the intent was to increase capacity in internal eva luation and develop frameworks to
evaluate core programs. However, as the group went through the process, the discussion and work often focused
more on questioning the design and delivery of individual programs and provided insight into management and
stakeholder expectations. The value to undertaking the evaluation internally by the management team will be the
focus of the presentation. The work of Arnold Love on the use of internal evaluation will frame the discussion. Clean
Foundation's experience will be used as an example of how internal evaluation influences program management and
the successes and challenges that have resulted.
The Role of African VOPEs in Influencing Gender and Equity Focused Evaluation by Governments
Jennifer Mutua
EXPOSÉ/PAPER
Africa is going through a transformation of hope and despair in equal measure. The landscape is characterized by
improved GDP growth rates coupled with new discoveries of natural wealth in some countries. Conversely, the
continent is rife with alarming increases in poverty and youth unemployment in an environment where economies
are not managed prudently. Waves of insecurity and patriarchal social order compound these. Unprecedented M&E
capacity building, including from gender and equity perspectives have b een offered to individuals and governments
by international development partners. Hence, a critical mass of professionals who believe in evidence-based and
equity focused decision making and implementation is slowly growing. However, national culture and p ractice
remain weak. Equity focused evaluation can influence more prudent public resource management through advocacy
leveraging on 2015 EvalYear spearheaded by Voluntary Organizations for Professional Evaluation (VOPEs).
The Role of Evaluation in the Scale up of Population Health Interventions: The Innovation Strategy
Leslie Payne
EXPOSÉ/PAPER
The Public Health Agency of Canada's Innovation Strategy (IS) supports projects to take action to reduce health
inequalities and effectively address priority complex public health problems and their underlying factors. The IS uses
an intervention research approach, meaning that the program supports the delivery a set of population health
interventions while generating critical knowledge, on (1) how the intervention pr ocess brings about change and (2)
the context in which the intervention worked best and for which populations. The IS uses an innovation 3 phase
model to assess and support the scale up interventions. This presentation will provide an overview of how the I S
model has used evaluation and intervention research to support the scale up effective population health
15
interventions in Canada and highlight a new approach to measure scale up readiness among population health
interventions.
To screen or not to screen: the impact of cancer screening letters
Emily Tsoa
EXPOSÉ/PAPER
Cancer Care Ontario's Evaluation & Reporting Team supports the Cancer Screening unit in evaluating the impact of
their three screening programs. These organized screening programs include the O ntario Breast Screening Program,
the Ontario Cervical Screening Program, and the Colon Cancer Check Program. Correspondence letters are currently
being sent to the general population who are due for screening. The impact of cervical screening invitation an d recall
letters, and colon cancer screening invitation letters, have been evaluated. Randomized control groups and historical
control groups were the designs used for these evaluations. Results from these evaluations will inform how future
screening correspondence should be delivered to the residents of Ontario.
Putting a ring on it: Evaluating the long-term engagement of operational staff
Joseph Travers
How does a small evaluation group engage a large operational group of staff and management over a multi-year
project to ensure the goals of the evaluation are met? The Ignite presentation will cover feedback we've received
from staff and management about how we've done so far in doing this, and wh ere we need to improve.
Achievements, failures, and unexpected situations will be covered, as well as how to use this feedback to enhance
the operationalization of this project going forward.
13h15-15h15
Avons-nous réglé la question de l'utilisation des résultats de l'évaluation?
Marie Gervais, Francois Dumaine, Olivier Sossa, Andrealisa Belzer, Courtney Amo, Anne Routhier,
Mirianaud Oswald Agbadome
PANEL/PANEL
Au Canada, des efforts importants ont été investis depuis plus de 10 ans pour améliorer la qualité de l'évaluation et
renforcer la capacité en évaluation aux différents paliers gouvernementaux. Différents chemins ont été pris pour
convaincre les gestionnaires et les décideurs de la valeur ajoutée de l'évaluation à l'élaboration et à la gestion d es
politiques et programmes publics. Toutefois au quotidien, quel sort réserve -t-on aux résultats des évaluations? En
quoi ces résultats viennent-ils véritablement éclairer la prise de décision? L'évaluateur pourrait -il faire mieux dans sa
capacité à mobiliser autour des résultats d'une évaluation? Ce panel réunira des experts provenant de divers
contextes de pratique (gouvernement, secteur privé, milieu académique) et de différentes régions du Canada. Ceux ci partageront leurs perspectives sur les enjeux résiduels de cet aspect de la pratique.
14h45-16h15
APPRENDRE ENSEMBLE: L'IMPORTANCE DES COMMUNAUTÉS DE PRATIQUES EN ÉVALUATION
Raïmi B. Osseni, Erika Kaneza
Au printemps 2014, quelques étudiants de l'école nationale d'administration publique décident de se retrouver
régulièrement pour échanger au sujet des pratiques et du secteur de l'évaluation en mettant l'accent sur la région
de l'Outaouais. Peu à peu, cette communauté de pratique va se développer et offr ir à ses membres un espace
permettant d'approfondir leurs apprentissages et de renforcer l'accès et le partage des connaissances. Les cercles
d'apprentissage sont des « petites communautés d'apprenants qui se regroupent intentionnellement, dans le but de
se soutenir dans le processus d'apprentissage » (Collay, Dunlap, Enloe et Gagnon, 1998). La présentation de l'équipe
16
d'évalue-action propose d'explorer son modèle de développement professionnel et ceux qui l'ont inspiré, ses
réalisations, les leçons apprises, et de découvrir les avantages de créer de tels cercles au sein de milieux
professionnels.
Building Capacity in Community-Based Networks: Partnership Grants Learning Project
Heidi Schaeffer
EXPOSÉ/PAPER
How can we know more about the difference that investments in building capacity with not-for-profit networks and
partnerships can make? Health Nexus and the Association of Ontario Health Centres, working with the Tamarack
Institute and with leadership from The Mowat Centre for Policy Innovation at the University of Toronto, asked just
this question in a recent evaluation. Come and hear about the Partnership Grants Learning Project that set out to
measure outcomes of a 7.2 million dollar investment in 27 not for profit community organizations by the Onta rio
Ministry of Citizenship and Immigration. An innovative new tool called Outcome Harvesting was used. Learn about
how Outcome Harvesting, inspired by Outcome Mapping, can help practitioners operating in networks and
partnerships to evaluate the social change results they are achieving through a learning oriented participatory
evaluation combining Outcome Harvesting with Most Significant Change and Social Network Analysis.
Causality, Complexity, Critical Realism, Validity: What Do These Terms Have in Common?
France Gagnon
It is a complex world out there. A program's causal pathway to change is rarely linear and simple. Different causal
pathways will exert a differential impact on various members of the target popu lation under similar implementation
conditions, settings, and times. Or, the program cultural, socio -economic, political context may negatively impact
outcomes in one setting but not another. And evaluation findings will have different meaning, relevance, and utility
depending on whether one is a program developer, program intervention manager, staff, community member, policy
analyst at the provincial or federal level or member of the target audience. How is one to produce evaluations that
are more credible and actionable? This presentation will review and present literature on causality, complexity,
critical realism and validity to provide suggestions for improving the construct validity of interpretations.
Contribution d’Avenir d’enfants dans le renforcement des capacités de ses partenaires en évaluation
Julie Rocheleau, Youssef Slimani
EXPOSÉ/PAPER
Depuis 5 ans, Avenir d’enfants (AE) soutient financièrement plus de 130 regroupements locaux de partenaires (RLP)
sur l’'ensemble du territoire québécois. AE est une organisation apprenante qui a le privilège de chapeauter un
projet innovant et mobilisateur en faveur du développement global de la petite enfance. AE note un renforcement
de la culture évaluative auprès des partenaires engagés dans les projets qu ’il soutient. AE a misé sur le renforcement
des capacités de ses partenaires, ce qui a mené petit à petit à une adhésion aux processus évaluatifs. Ceci s ’est
traduit par le développement de nouvelles compétences, et l’intérêt croissant démontré à l’égard de l’évaluation.
Cette présentation mettra en évidence, à travers des expériences du terrain et des exemples au niveau
organisationnel, le chemin qui a été parcouru par AE et ses partenaires dans le but de développer et consolider une
culture évaluative dans le domaine de la petite enfance.
Data collection and their methods for Impact Evaluation
Raed Zahrawi
A five-year Compact ($275 million) is developed to reduce poverty and increase income in Zarqa Governorate of
Jordan, through increases in the supply of water available to households and enterprises through improvements in
the efficiency of water delivery, the extension of wastewater collection, and the expansion of wastewater treatment.
Measuring the impact of the Compact activities on economic and social outcomes was designed to establish a causal
17
relationship between program interventions and observed changes in household availability and consumption of
different sources of water, household income, and household expenditure indicators. For the purposes of
presentation, complementary data collection activities (Household, Enterprise, and farmers surveys) will be outlined
over the Impact Evaluation components, with their methods, as well as dem onstrating the use of the exiting country
systems and the introduction of technology in data collection.
Determining an appropriate range: Calibrating Evaluations
Beate Schiffer-Graham, Yves Gingras, Arvind Srivastava
PANEL/PANEL
Within the current federal government context of fiscal restraint and an ever increasing organizational tempo,
federal evaluators need to identify approaches that are nimble and flexible while being true to the fundamental
standards of evaluation. An opportunity that merits further discussion is the art of “calibration”, frequently used in
the world of weights and measures. The activity of “calibration” generally seeks to determine an appropriate range
to ensure that the tool or instrument conforms to a standard. For the purpose of this presentation 'calibration”
refers to evaluators decreasing or increasing the range of 1) methodologies and/or 2) scope while complying with
the standards for federal evaluations. Calibration, if done well, promises to be a partial support in providing senior
management with more timely evaluation findings at lower cost.
Developing an Evaluation Framework for Research-Oriented Communities of Practice
Kaileah McKellar, Johanne Saint-Charles, Whitney Berta, Donald C Cole, Rhonda Cockerill
Communities of Practice (CoPs) are increasingly used in diverse sectors; however there is a lack of empirical
evidence of how they work. To address this gap we have developed an evaluation framework to guide evaluations of
research-oriented CoPs. To develop the framework, we used data on the context, mechanisms, and outcomes of
CoPs, gathered from a systematic scoping review that drew from databases in health, business, and interdisciplinary
fields, as well as key informant interviews with CoP members. The framework includes suggested approaches,
measures, methods as well as a program theory. The presentation will outline methods used in developing the
framework, highlight the key mechanisms that lead to social learning and act ion, discuss the level of evidence of key
mechanisms and offer a series of indicators aligned to the program theory. The presentation will conclude with a
proposed approach to testing the evaluation framework, including key mechanisms, in different context s.
Developing indicators to facilitate evaluation of Age-Friendly Communities initiatives across Canada
Kara Haynes
EXPOSÉ/PAPER
Public Health Agency of Canada developed an evaluation guide Using Indicators in Your Age -Friendly Community
Initiative, for over 900 communities in Canada that are implementing Age-Friendly Communities Initiative (AFC). The
objective of this project is to facilitate the uptake of evaluation by the communities implementing AFC initiatives.
Hence this resource targets community organisations, seniors, municipal officials, program planners and policy
makers responsible for implementing AFC. The guide provides an introduction on evaluation, and how to design an
evaluation framework using logic models. Since there is a large variation in the way AFC initiatives are implemented,
this guide provides flexibility to choose from a menu of indicators relevant to the programs, and relevant assessment
tools both qualitative and quantitative.
Evaluating Complex Programs
Jacques Bérard
EXPOSÉ/PAPER
Government and philanthropic organizations that aim at producing systemic change work at many levels, with many
actors using multiple strategies. Think of the UN's Millennium Goals. Programs and policies that address complex
systems interacting on horizontal planes, and vertically at national, sub-national and local levels pose a challenge to
18
evaluators. How does one evaluate the impact, or the performance, of a program when so many other partners are
working towards (or against) the transformation that is sought after? This presentation will address some of the
challenges that evaluators face in dealing with complex programs. It will attempt to provide elements of solutions by
looking at monitoring and evaluation approaches and methods borrowed from the field of complex adaptive systems
and developmental evaluation.
Evaluating research team processes for better research implementation
Susan Roelofs, Cody Anderson
EXPOSÉ/PAPER
A Canadian team leads the evaluation of the five-year European Commission-funded REPOPA Project (Research into
Policy to Enhance Physical Activity), which studies the integration of research evidence with real -life policymaking in
six European countries: Denmark, Finland, Italy, the Netherlands, Romania, and the United Kingdom. Annual process
evaluations use mixed methods with five distinct tools: a collaboration survey; a social network mapping survey;
document review; a junior researcher research competency self-assessment; and semi-structured focus groups. The
participatory, utilization-oriented evaluation incorporates innovative strategies to stimulate the use of findings for
continuous improvement and consortium decision-making. Consortium members provided input into evaluation
guiding principles and design; into internal monitoring reports; and through targeted discussion at annual meetings.
Two cycles of process evaluation have been conducted thus far.
Évaluation développementale d'une plateforme web pour le transfert de connaissances
Esther Mc Sween-Cadieux, Christian Dagenais
La Fondation du Dr Julien travaille actuellement au développement d'une plateforme web pour le transfert de
connaissances dans le but de faire connaitre et reconnaitre leur modèle de pédiatrie sociale en communauté. Une
section importante de cette plate-forme est dédiée aux divers intervenants des centres de pédiatrie sociale partout
dans la province pour les aider à améliorer leurs pratiques via des modules de formation en ligne. L'approche
d'évaluation développementale (Patton, 2011) encore peu utilisée, permettra de suivre pas -à-pas le développement
de la plateforme en soutenant la Fondation du Dr Julien à faire certains choix durant le processus de création et
d'expérimentation des modules. En se basant sur des entretiens avec les acteurs impliqués dans le processus
d'évaluation et sur notre propre expérience, les conditions de succès de cette approche pour faciliter le
développement d'une stratégie de transfert de connaissances seront énoncées.
Evaluation Standards in the Spanish Speaking World: Evolución y desafíos!
Mariane Arsenault, Esteban Tapella, Pablo Rodriguez- Bilella, Maria Bustelo
PANEL/PANEL
The development of evaluation standards has been considered an indicator of the increasing relevance and
professionalization of the evaluation practice. This panel will introduce a general framework on the usefulness of
evaluation standards in order to strengthen an evaluation culture, with illustrations drawn from the experience of
Spain and the Latin America region. A particular focus of this panel will be on explaining the role played by Voluntary
Organizations for Professional Evaluation (VOPEs) in the development of standards, and the main challenges faced in
this endeavour. The panel discussions will take place in Spanish; this is a first at a CES Conference and is highly
coherent within the context of the 2015 Year of Evaluation.
Intervenir en périnatalité et en petite enfance pour améliorer la santé et le bien-être des familles
Nathalie Dubois,
EXPOSÉ/PAPER
Intervenir en périnatalité et en petite enfance pour améliorer la santé et le bien -être des familles vulnérables:
Évaluation de la mise en oeuvre et des effets du projet de La Maisob Bleue. Depuis 2007,La Maison Bleue offre des
services en périnatalité et petite enfance dans le but d’aider les femmes et leur famille à accueillir leur bébé. Face au
19
développement grandissant de La Maison Bleue et aux multiples demandes formulées par divers acteurs quant à la
possibilité de répliquer le projet dans d’autres contextes et milieux, une recherche évaluative visant la production de
connaissances crédibles et valables s’est avérée nécessaire. Le présent rapport de recherche vise à répondre à ce
besoin. Il a comme objectif d'appréhender le modèle d’intervention privilégié par La Maison Bleue, d'analyser sa
mise en oeuvre et d'évaluer ses effets.
Is it time to ditch the deck?
John Burrett
John will argue that the briefing deck is the most ineffective method possible for communicating findings and
briefing senior officials. He will back this up with an analysis by Edward Tufte of how the reliance on PowerPoint
briefings led to the poor decision-making behind the Columbia space shuttle disaster in 2003. The presentation will
finish with suggestions being advanced by communications experts for better briefings.
Lancement officiel de la "trousse de renforcement institutionnel des VOPEs » (VOPE toolkit)
Marie Gervais, Jim Rugh, Ziad Moussa
LANCEMENT/LAUNCH
Networking to navigate the Value for Money waters: An experience of DFID-Nigeria VFM forum.
Oluwatosin Akomolafe
Value for Money has been around for awhile but has lately been a requir ement by donors. It has now become
imperative that value for money is shown from the bidding/tendering process through the project cycle to impact. At
times, programmes/projects do not understand the VFM requirements of the donors; have conflicting underst anding
of requirements and approach to mainstreaming, assessing and reporting VFM. Also, Project/Programmes at
different phases of implementation have encountered different challenges in showing value for money. New
programmes are particularly concerned about how to start, while older ones are uncertain of how to introduce VFM
mid-way into their projects. Networks have been found to be useful source for building capacity especially for
beginners where everyone can learn in a non-threatening environment and leverage resources amongst others. The
DFID-Nigeria VFM forum is a network of DFID funded programmes and it is aimed at supporting members.
Obtaining young children's assent to participate in research
Nick Petten
The methodology of research is critical in determining any truth claim that a research study posits. In researching
childhood, a critical methodological concern is the power dynamics that occur between adults (who are typically the
researcher) and children (who are typically the subject). In order to make any truth claims based on the scientific
process, careful consideration must be given to the ethics of involving children in research so that power dynamics
can become more equalized. This process begins with obtaining children's assent to participate in the research
process. As an adult researcher, the process of obtaining children's assent rests on the recognition that children are
competent social actors who are capable of making sense of and affecting their societies. From this recognition,
adult and child researchers can begin the design and sense-making activities of research that will enhance any
discoveries made about childhood.
Paternalists internationaux: Défis et retombées
Marie Gervais, Jim Rugh, Serge Eric Yakeu, Ziad Moussa, Ian Hopwood, Fation Luli
PANEL/PANEL
Avec l'appui de EvalPartners, divers partenariats ont été développés depuis 2013 entre sociétés nationales
d'évaluation actives sur plusieurs continents dans le cadre des concours Peer-to-Peer et Innovation Challenge.
20
Comment ces partenariats ont-ils été développés ? Quels étaient les objectifs recherchés? Quels défis ont été
rencontrés? Quelles retombées et quels apprentissages peuvent maintenant être dégagés? Quelques représentants
partageront leurs expériences et expliqueront comment ces partenariats ont contribué à renforcer les capacités
institutionnelles de leurs sociétés et à soutenir la réalisation de leur mission, incluant la promotion de l'évaluation
au pays. Dans le cadre de 2015 Année internationale de l'évaluation, voilà des exemples probants de l'effet
catalyseur de la mutualisation d'expériences. Nos panélistes s'exprimeront finalement sur les priorités de l'Agenda
global de l'évaluation 2016-2020.
Process Evaluation of Antiretroviral Therapy Program in Southwest Ethiopia: a case study
Fasil Walelign Fentaye
EXPOSÉ/PAPER
The evaluation assessed the implementation fidelity of ART program at Sekoru Health Center, Southwest Ethiopia.
Case study design with mixed methods was employed from June 1 to July 30, 2012. Client-exit interview, document
review, resource inventory and direct observation of provider-client interactions were the data collection
techniques. Criteria set together with the program stakeholders were used to judge the implementation. Fidelity of
the program's implementation was 61.3%. Adherence to the guideline was 54.6%; quality was maintained for 73.7%
of the services delivered; and clients' responsiveness was 83.2%. Implementation fidelity of t he program was low
despite higher participant responsiveness and enabling contextual factors. ART services at the health center are in
need of urgent improvement and the evaluation framework showed program implementation from provider &
participants perspective.
Process, Summative, Impact or something else: a hybrid approach of evaluation in non-profit sector
Faisal Islam
EXPOSÉ/PAPER
Amidst growing demands on non-profit organizations to show the evidence of their outcomes, there has been
always desire to conduct evaluation to demonstrate the results. But many NGOs do not have the luxury to wait until
the projects end and then plan for impact evaluation. Several pressures warrant NGOs to conduct evaluation early —
while the projects are still being implemented —to know the results. With rapidly changing priorities and low
resources dedicated to M&E, NGOs to conduct evaluation that can serve all purpose at once: show the impact;
improve the processes; fulfill the donor requirement; scale up the projects; & bett er understand the needs of
beneficiaries. Drawing on a recent experience of a multi-country evaluation, this paper attempts to explore how a
hybrid approach to evaluation, using the elements of ‘utilization -focused’ and ‘real world’ evaluation, can provide a
framework of evaluation for multiple audiences and a variety of objectives for NGOs & Civil Society.
The Impact of Gaucho Winter Program on Older Adults Respiratory Infection in Southern Brazil
Marilia Ramos
EXPOSÉ/PAPER
This article presents results of the evaluation of a project developed by Rio Grande Health Secretary: the Inverno
Gaucho Project, which intends to reduce the hospitalization of older adults due to respiratory infection. The
evaluation involved counties that participated in the project in comparison to those counties that didn't participate,
to control for characteristics that can affect the hospitalization due to respiratory infection rates (dependent
variable) that are not related to the project. Descriptive analysis, multivariate regre ssion models and the propensity
score matching techniques were used. It was observed that the greatest mean difference among participants and
non participants happened during the year 2008. Participants presented lower hospitalization than non participants .
Useful Theories of Change: Purposefully Building Reach into Program Theory
Steve Montague, John Mayne
PANEL/PANEL
21
This coordinated panel will suggest the need for theories of change that can be actually used for planning and
managing interventions and their evaluations, especially the need to build reach (individuals, groups and institutions
a program or initiative engages with to achieve mission) into the logic models, program theories and results
frameworks of essentially all initiatives. The panelists make the case that theories of change can and should be
practical tools and that theories of reach should be considered part of program theory. The point is made that a lack
of explicit thinking about reach in theories of change and logic models can lead t o problems such as
narrow/constricted understanding of results chains, favouring 'narrow and efficient' initiatives over 'wide and
engaging' initiatives, taking for granted intermediaries co-delivering an initiative and creating a bias against equity
considerations. Audience engagement will be encouraged in this session.
Using evaluation to improve emergency response to homeless persons who abuse substances in Montreal
Kristelle Alunni-Menichini
Homeless people who abuse substances are facing multiple challenges on their trajectories of health care services.
Our objective is to identify ways that would improve the emergency response by an approach that rally all keys
actors (i.e. experts committee, health providers, community stakeholders, police officers and homelessness persons
with substance disorder). We choose a research design inspired from the deliberative democratic, participative and
empowerment evaluations. It will allow us to engage participants in several moments of exchange in order to
disseminate credible/objective information, to develop a common understanding, and to discuss and validate our
findings. More precisely, the actors' needs and several interventions will be evaluated. Then, these interventions will
be prioritized according the appropriateness, the actors' needs and the context. Finally, the dissemination of results
will be conducted among the keys actors and the community.
Using program evaluation to impact health human resource policy development
Kathryn Hodwitz, Wendy Yen, Joseph Travers
EXPOSÉ/PAPER
The College of Physicians and Surgeons of Ontario (CPSO) regulates the practice of medicine by providing licenses to
physicians and monitoring standards of practice. Over the last 15 years, the CPSO has developed numerous
registration pathways and policies to reduce physician shortages and increase access to licensure. In 2012, the
College embarked on a program evaluation of its registration pathways to ensure that the policies are meeting their
intended purposes, namely facilitating entry to the profession for qualified doctors. The evaluation seeks to
determine if performance differences exist between physicians registered through the traditional pathway (those
who completed post-graduate training and examinations in Canada) and alternative pathways (all other physician
applicants). The methodological challenges and lessons learned for a complex evaluation will be presented, as well
as the ways in which the evaluation data will inform future policy development.
Walking the Walk: Building Evaluation Capacity through a Community of Learning, Inquiry and Practice
Chris Lovato, Derek Wilson, Kylie Hutchinson, Beverly Parsons
EXPOSÉ/PAPER
Improving staff knowledge and skills in evaluation can contribute significantly to creating a culture of evaluation that
will facilitate success. However, organizations are often challenged by limited resources for evaluation capacity building. Communities of Learning, Inquiry, and Practice (CLIPs) is a simple, effective, and cost-efficient model that
contributes to evaluation capacity building by increasing the knowledge, skills, and understanding of evaluation by
individuals within an organization. This presentation will describe the implementation and evaluation of a CLIPs
process in a university medical education setting. We will share our experience and insights about how the CLIPs
process can successfully build and support a culture of evaluation.
Walking together to make a path: developmental evaluation with a non-profit women's organization
Alma Estable, Mechthild Meyer, Raine Liliefeldt
22
We'd like to share the story of what happens when a small evaluation firm committed to social chang e works with a
national non-profit women's organization in a developmental evaluation process- what we learned, what we forgot,
what we wish we had done, what we managed to achieve, and where we ended up. Why? Because others might
learn from an approach to evaluation that attempts to combine, in a very practical way, the diverse skills, expertise,
reach, and mandates of a team of evaluators with that of a team of feminist organization workers , who all share a
common vision for a more just and equal society.
Working with Assets to Build Capacity: Learning Circles and Appreciative Inquiry in Continuing Care
Gail Vallance Barrington, Sandra Woodhead Lyons, Sharla King
EXPOSÉ/PAPER
The learning circle is a capacity-building activity based on well known concepts in adult learning, reflective learning,
and quality circles but the impact of this low-cost learning model has not been well studied. A 2013 pilot in three
continuing care sites in the Calgary area was the base for the current study of 13 learning circles in seven continuing
care organizations across Alberta. The pilot evaluation was grounded in Appreciative Inquiry (AI) (Preskill and
Catsambas, 2006) and while small in scope, it documented evidence of knowledge transfer and practice change.
However, as Alschult (2014) asked recently, how can we embed a mindset in an organization or community to utilize
its strengths for growth and improvement? Scaling up the evaluation processes from the pilot for this larger study
has provided some unique issues and challenges but AI continues to prove its utility in terms of understanding
emergent change. Some study tools and early observations will be shared.
Innovative approaches for dialogue: Photovoice and World Café / Dialoguer: Les approches innovantes du
Photovoix et du World Café
Photovoice
Christine Loignon, Karoline Truchon, Chris Corrigan, Amy Lenzo, Moderated By Huilan Krenn, W.K. Kellogg
Foundation
PLÉNIÈRE/PLENARY
15h15-16h15
Benefits and Limitations of SROI Analysis
Anne Miller
DÉBAT/DEBATE
Social Return on Investment (SROI) is a growing evaluation methodology that seeks to move the value assessment
process from a focus on cost alone to one that focuses on social value creation from a stakeholder perspective. SROI
assigns financial proxies (monetary value) to social, environmental and economic outcomes, to try to understand the
value of achieving outcomes as compared to the cost of investing in an initiative. While SROI creates interesting new
information about many projects, it is not necessarily the best evaluation method for every situation. This session
will explore the benefits and limitations of the SROI approach, fostering a discussion on where this methodology fits
within the Canadian evaluation context.
16h15-16h35
Navigating Hostile Waters: Using Evaluation to Achieve Rapprochement
Cynthia Tananis
EXPOSÉ/PAPER
The context: Closure of two (K-8) community schools and reopening one as a combined facility. The issues: How do
you honor the two communities (each with a strong identity) and each school with long histories and experienced
staff --- and merge them to a new entity? The problem: The merger occurred in a top-down hierarchical way with
23
minimal consideration of the complex issues. This paper describes how external "evaluation" was used to create
space for shared values and shared decision-making after the damage had been done. Explore how evaluation can be
used as a healing force by finding common ground, shared values and vision, and a vision to guide action.
16h35-17h35
All Things Equal? A Health Equity Lens for Evaluation
Ashley Zelmer, Kelly De Cecco
EXPOSÉ/PAPER
In this presentation, participants will learn about the Health Equity Lens for Evaluation, a tool developed by the
Office of Evaluation at the Public Health Agency of Canada and Health Canada to help ensure that health equity
considerations are taken into account by programs. The presentation will focus on the development of the tool, how
it can be applied through the evaluation process, in particular as the issues of relevance and pe rformance are
explored, and how it can be used to help guide decision-making. The presentation will also touch on the importance
of addressing health equity concerns in order to help create the world we want; a world with healthier Canadians.
This promising tool helps evaluators identify areas of health equity concern, which can lead to improved programs
that impact the health of all Canadians, including those most at risk.
Analyse d'un processus délibératif pour renforcer la politique de gratuité des soins au Niger
Ibrahim Hamani Souley
En 2006, le Niger a mis en place une politique de gratuité des soins pour les femmes enceintes et les enfants de 0 à 5
ans. En 2012, un processus délibératif fut organisé pour échanger sur les acquis, limites et perspectives de cette
politique. L'objectif de cette recherche est de comprendre les effets du processus ainsi que d'explorer les activités
du comité de suivi de la feuille de route. La recherche a été réalisée en 2014 à Niamey. Elle a reposé sur l'utilisation
du cadre conceptuel de Boyko et al., (2012) avec un accent particulier sur les trois formes d'utilisation des
connaissances présentées Dagenais et al., (2013): instrumentale, conceptuelle, persuasive. Des entretiens semidirectifs ont été effectués. Les résultats révèlent une utilisation plus instrumentale des recommandations et une
utilisation conceptuelle et persuasive à un degré plus faible. Le comité de suivi de la feuille route de la conférence
n'a pratiquement pas fonctionné.
Assessing and attributing research impacts using novel approaches: an update
Christopher Manuel
EXPOSÉ/PAPER
A pilot study was undertaken in 2014 to examine the utility and feasibility of adopting an independent, rigorous and
replicable methodology for identifying the influence research supported by the Canadian Institutes of Health
Research has had on decision making in the United States. We entered into a collaboration with the American
National Guidelines Clearinghouse to match approximately 10,000 CIHR supported articles from 2008-2010 with the
scientific literature cited in the clinical guidelines and quality measures (and supporting documentation) included in
the NGC databases. Matches were then reviewed to determine the strength of the influence the CIHR supported
research had on the development of the guideline/quality measure. This presentation will describe the methodology
used, the results and discuss lessons learned.
Assessing the Economic Impact of Snowmobiling in Ontario - Different Models / Approaches
Harry Cummings, Shannon McIntyre, Don Murray
EXPOSÉ/PAPER
The Ontario Federation of Snowmobile Clubs (OFSC) promotes snowmobiling experiences that are safe, enjoyable
and environmentally sustainable. More than 200 community-based OFSC member snowmobile clubs operate more
than 30,000 km of signed and groomed trails for use by approximately 100,000 snowmobile trail permit holders and
24
their families. Snowmobiling generates significant economic benefits in Ontario and these impacts ha ve been
examined several times over the last 15 years using three different models / approaches. The most recent study
completed in 2014 utilized the Tourism Regional Economic Impact Model (TREIM) in conjunction with an online
survey of over 6,000 snowmobilers to provide the most complete assessment of snowmobiling activity in Ontario to
date. This session will compare and contrast the different models used to assess the economic impact of
snowmobiling in Ontario.
Assessing the reach of information and R&D services using social network analysis
Adele Acheson, John Burrett
EXPOSÉ/PAPER
As the globalization of information increases, so does the importance of looking at how information and research
products are disseminated. This is a critical issue in assessing the effectiveness of public R&D. Network analysis has
existed as a methodology for some time in fields such as sociology, anthropology and biology, providing a rigorous
means to understand how people, organizations, organisms, and so on are connected. More recently its application
has expanded to the analysis of networks of communications such as in social media. This technique has seen little
use in evaluation to date, but we contend that as both the scale and potential for transmission of information as a
product of public research grows, its attractiveness as an evaluative methodology becomes more and more
apparent. This presentation discusses the use of social network analysis to evaluate the reach and impact of
research and knowledge, undertaken by the National Research Council Canada.
Assessment of stakeholders participation in local government project evaluation: a case study of saw
Nurudeen Mohammed Aliu
The study entails a systematic examination of the actual level of primary stakeholders' involvement in project
evaluation and comparing it with the actual evaluation practices in the district. It also investigated the factor
influencing participatory evaluation in the district. The communities studied inclu de Sawla, Tuna and Gindabour with
a sample of 40 primary stakeholders selected through proportional random sampling method. The study was
facilitated through a desk review of available literature and the employment of Participatory Rural Appraisal (PRA)
tools which included key informant interviews, focus group discuss and household questionnaire interview were
employed for this study. The data were analyzed by the use of descriptive statistics such as frequency and cross tabulation using Microsoft excel. Dependent and independent variables such as institutional arrangement, planning
process and evaluation process were critically reviewed
Bottoms up or top down: successes and challenges in M&E of large policy frameworks
Jennifer Young, Katy Pollock, Bridget Hall, Amanda White
EXPOSÉ/PAPER
Large policy frameworks often drive service and program delivery for end users. A recommended practice is to
attach a monitoring, evaluation and accountability (MEA) process to the policy framework, in addition to program specific MEA processes to the programs under the policy. This facilitates the use of data for program-level and
umbrella policy-level monitoring and evaluation purposes. A common question arises with whether to develop the
policy-level MEA framework first, or to begin by developing MEA plans for individual programs. The former
establishes clear guidelines for all programs early in the process, enabling easier "roll -up" of data, while the latter
ensures that the policy framework will be relevant to the specific needs of the individual programs. We will discuss
the successes and challenges of both approaches using 3 NWT examples, and provide recommendations for users
involved in the monitoring and evaluation of large policy frameworks.
Comment évaluer la collaboration et la cohésion des parties prenantes ?
Eva Anstett, Vénétia Sauvain
EXPOSÉ/PAPER
25
Plusieurs interventions publiques rassemblent plus d'un acteur agissant sur une problématique donnée (plusieurs
organisations publiques, milieu communautaire, secteur privé, organisations citoyennes...). L'objet de ces
interventions publiques consiste souvent à faire travailler ces personnes ensemble afin d'améliorer l'efficacité de
leurs actions sur le terrain. Dans un contexte de rareté des ressources, où on che rche de plus en plus à faire mieux
avec moins de moyens, il est important que l'on puisse évaluer si une intervention publique peut permettre une
meilleure cohésion entre différents acteurs ce qui peut aussi avoir ultimement un impact sur les cibles finale s des
actions. Nous avons utilisé une méthode pour évaluer cette cohésion dans plusieurs de nos dossiers (échelle de
Metcalfe) et souhaitons partager cette méthode tout en étudiant ses avantages et ses inconvénients.
Complementary instrument for the identification of proxy indicators for research & training programs
Véronique Dugas
EXPOSÉ/PAPER
Mitacs is a not-for-profit organization promoting innovation through the delivery of research internship and training
programs in Canada. As with many innovation-focused programs, some Mitacs program outcomes are diffuse, slowacting and multi-dimensional. Recently, Mitacs has integrated the use of longitudinal studies to complement the
information captured through its Accelerate program exit surveys. Th is strategy has proven useful as a means to
complement the data obtained shortly after the end of the internship and to better define long -term impacts of the
program. Of interest, comparison of data collected through program exit surveys vs. longitudinal questionnaires also
allows for the identification of appropriate short-term proxies that go beyond usual research outputs. In conclusion,
the use of distinct instruments used at different time points allows for a more complete picture of Accelerate's
outcomes for its participants and the broader research and innovation ecosystem.
Connecting Evaluation and Policy Imperatives: Mitacs as a Case Study.
Valerie Walker
EXPOSÉ/PAPER
Promoting innovation is a significant focus for governments seeking to improve productivity and growth in a
competitive global economy. As such, programs that promote and support innovation are key parts of government
economic policy. Through its various programs, Mitacs has become an integral part of Canadian innovation policy
during the last fifteen years. Its remarkable growth has largely been driven by effective use of evaluation to both
direct program design and to ensure effective alignment with broader policy objectives. This presentation will
describe Mitacs' success in aligning innovative program evaluation with the broader policy objectives of government
funders and to support timely evidence-based government decision-making. It will focus on how we built a
reputation as credible, collaborative, and innovative by using output s and outcomes to drive program improvements
while simultaneously mastering how to effectively communicate impacts to government decision makers.
EVALUATION DES CAPACITES DES ORGANISATIONS LOCALES REPERTORIEES DANS LE CADRE DU PROJET DPP
Jean Paul Ngantchou Nkwangue
EXPOSÉ/PAPER
Le partenariat public-privé est devenu au fil du temps un instrument additionnel de financement du développement
durable. La GIZ, Coopération allemande pour le développement en partenariat avec la société AES - SONEL, ont mis
sur pied cette forme de coopération dont l’objectif est d’améliorer les conditions socio économiques des
populations vivant dans les zones situées à proximité du plus important barrage hydroélectrique exploité au
Cameroun. Ce partenariat tourne autour d’un projet intitulé Partenariat pour le développement durable dans
l’arrondissement de Massok-Song Loulou et Pouma. Il a pour objectif d’apporter un appui technique aux
organisations économiques locales à la production et à la commercialisation de la banane plantai n. l'évaluation Des
études de faisabilité, ont démontré de manière générale, la caducité des organisations par la récurrence de certains
dysfonctionnements liés aux capacités organisationnelle, institutionnelle et opérationnelle.
26
Evaluation meets ancestral midwives: how real is evaluation getting with intercultural policy?
María Luisa Calderón
EXPOSÉ/PAPER
In the last 4 years in Guatemala, Healthy Maternity Law was published and took effect and Indigenous Midwives
inclusion policy is in the making due to maternal health becoming a matter of national urgency; the country has the
3rd highest maternal mortality in Latin America. The study analyses evaluation theory applied to policymaking of the
inclusion of indigenous midwifes from the four nations in Guatemala: Mayan, Garifuna (African descendents), Xinca
and Mestizo (mixed); as well as its challenges and opportunities.
Evaluation of the Ivirtivik Centre - Lessons Learned from Learning Together
Donald Murrary, Shannon McIntyre, Valérie Roy, Loïc Lenen
EXPOSÉ/PAPER
Inuit in the South face challenges which create significant barriers to employment (Aboriginal Skills and Training
Strategic Investment Fund, 2013). The Ivirtivik Centre is an employability and skills development initiative for Inuit in
Montreal. An evaluation of the Centre found that the program has made progress and continues to evolve from a
partnership of organizations that worked to introduce a structured program that encourages progressive
development of employability skills while being sensitive to the challenges faced by Inuit. The Centre has helped
participants build their self-confidence and connect them to the community through various activities that combine
their values, talents and preferences. The Centre has positively changed the parti cipants' employability and their
quality of life. The presentation focuses on the evaluation of the Centre and its contribution to the world we want
for the Inuit living in the South.
Facilitators and Barriers of Patient and Family Engagement in Health Service Improvement
Nathalie Gilbert, J. Bradley Cousins
EXPOSÉ/PAPER
The underlying philosophy surrounding patient-centred care advocates for patients and family members to have an
active role in all areas of their care, including broader areas of the health care system such as in evaluation. Despite
recent efforts in the past few years that would see greater patient and family engagement in health service
improvement initiatives, there remains a lack of best practices and understanding about the process of en gaging
patients and family members in evaluations within the health sector. The Ottawa Hospital Psychosocial Oncology
Program (PSOP) recently completed a program review and needs assessment with the PSOP Patient and Family
Engagement Committee; composed of patients, family members, a program manager, health professionals and an
evaluator. In conjunction with this program improvement initiative included a corresponding exploratory study
examining the process of patient and family engagement in a health servi ce improvement project. This study
employe
How to ensure the evaluability of the future Regional Public Health Plan before its planning begins?
Yassen Yordanov, Nathalie Dubois
EXPOSÉ/PAPER
Although increasing attention is paid to the planning of the futur e Regional Public Health Plan in Montreal (RPHP),
the question of its evaluability when this plan is still in phase of planning raise particular theoretical and
methodological issues. The objective of the presentation is to share the theoretical and method ological knowledge
acquired during this journey. First, published literature regarding evaluability assessment (EA) was reviewed to
propose a taxonomy of the most important purposes for EA and EA outcomes. Secondly, based upon the taxonomy
results, the presentation summarizes the methodological and strategic approach developed by the team. The
proposed approach suggests that it is possible to support a more systematic consideration of criteria for a
meaningful evaluation in linking past EA knowledge that might be generated from an EA to a further RPHP or policy
planning process. And finally, recommendations to enhance RPHP planning and meaningful evaluation are offered.
27
Impacts of evaluation on regulatory programs and its contribution to enhanced governance.
Perfecto Vélez Macho
EXPOSÉ/PAPER
Participants will be presented with an overview of changes stemming from observations and recommendations
linked to the evaluation of federal, health-related, regulatory programs. Continuous program engagement as an
approach leading to internal program reflection, discovery and enhanced evaluation results will be discussed in the
context of evaluation influence together with the role of evaluators and evaluation units as “knowledge
sharers/communicators” of evaluation results. The role of evaluation as a tool leading to improved program
understanding, enhancement and greater accountability will be discussed in the context of better governance.
Involving youth in a participatory and developmental evaluation
Nick Petten
EXPOSÉ/PAPER
Programs and services designed for youth have traditionally excluded the participation and voice of its main
stakeholders and beneficiaries. This often results in poorly designed interventions that further exclude youth and
misses opportunities for empowerment. A youth participatory and developmental evaluation was conducted for a
youth mentorship program designed for young community leaders that have started their own arts -based, social
enterprises in downtown Toronto. Youth in the program were encouraged to participate in the evaluation through
various methods including obtaining youth assent, reflective discussions, creating accountability systems, and
assuring confidentiality and appropriate use of youth voices. This paper presen ts the lessons learned using various
strategies and approaches to working with youth, as well as, the systematic and ethical strategies of maximizing
participation and appropriate use of evaluation findings.
Knowledge Transference in Evaluation Capacity Building: Is Client Intent Necessary?
Melissa Rogan
EXPOSÉ/PAPER
For fifteen years, the topic of evaluation capacity building (ECB) has received considerable attention in the valuation
literature (Boyle & Lemaire 1999, Leviton 2001, Preskill & Boyle 2008). Despite some variation in ECB models, one
consistent ECB tenet is it is always intentional, both on the part of the evaluator and the client. However, if the goal
of ECB is to transfer the knowledge, skills, and attitudes necessary for quality evaluation an d its use to the client, is
intent on the part of our clients necessary for that transference to occur? In this presentation a modified version of
ECB will be introduced, and the potential benefits for the discipline will be discussed.
L'évaluation économique d:une intervention complexe en santé publique
Lynda Benhadj
Dans un contexte de vieillissement de la population, de compressions budgétaires et de maîtrise des dépenses dans
le domaine de la santé et des services sociaux, la fonction d'évaluation économique des programmes prend tout son
sens. Dans le domaine de la santé publique, les programmes qui s'adressent aux populations vulnérables, sont
souvent qualifiés de « complexes » vus leurs effets multiples, diffus et parfois intangibles. L'évaluation économique
commence à s'intéresser à des outils d'analyse du rendement de tels programmes. Dans le cadre de ce travail
doctoral, nous utilisons une méthodologie adaptée pour évaluer la Maison bleue, une ressource offrant des services
de périnatalité sociale à des femmes enceintes cumulant plusieurs facteurs de vulnérabilité. Les objectifs poursuivis
seront: 1- décrire l'intervention complexe; 2- estimer les coûts de mise en place de cette intervention; évaluer les
conséquences; 4-analyser son rendement potentiel.
Not just stories: monitoring outcomes to inform program management and evaluation'
Marie-Caroline Badjeck, Pamela Kertland, Mary-Ann Wilson, Jennifer Ardiel
EXPOSÉ/PAPER
28
Evaluating program contributions to long term outcomes can be challenging and in the case of adaptation, additional
challenges include the long-time horizons needed for adaptation planning and implementation, and the cross sectoral nature of climate change impacts. The Climate Change Impact and Adaptation Division has adapted the
"Most Significant Change" and "Outcome Harvesting" approaches to collect stories of change focusing on 1) what
change happened, 2) when and 3) why. This paper presents how the two approaches we re implemented and offers a
range of insights on how to collect monitoring data that can contribute to impact evaluations as well as how to use
new technologies to better disseminate results. The experience presented in this paper illustrates the usefulness of
result-based monitoring at the program level, and the complexities associated with establishing causal relationships
for programs addressing cross-sectoral issues such as climate change.
Open program evaluation -- collective solutions to collective challenges
Andrew Dzuba
EXPOSÉ/PAPER
The world we want is one of increased transparency, greater citizen participation, and instantaneous adaptation to
change. Open program evaluation practice is the next step in participatory models of evaluation, in that i t brings
together all of the individuals, professionals and institutions interested in assessing a program -- disrupting the
current 'one-funder, one-program, one-evaluator' paradigm of program evaluation. By encouraging the development
of communities of practice leveraged by social media, open program evaluation practices can be created that
promote transparency and democratic engagement, thereby flattening disparities of social class and power that exist
between institutions, professionals, and program recipients. Ultimately, open program evaluation practices promote
collective solutions to collective challenges.
Peace, love, and evaluation: the pursuit of social justice
François Dumaine
As the field of program evaluation emerged, theorists came along to challenge practitioner in clarifying their
fundamental purpose. Is program evaluation largely limited to applying quantitative and qualitative methods in
order to better understand what a program or intervention is about? Or is there something more to it? Early
theorists such as Ernest House and, more recently, Jennifer Greene, have emerged to challenge evaluators to
embrace a path of social justice, including truth, beauty and justice. For Canadian evaluators, the fundamen tal
question around evaluation purpose remains highly relevant. Is evaluation about gathering and reporting on data
gathered through well-established methods, or is it about determining how evaluation can best support collective
goals related to social justice? Using current literature, the discussion will invite participants to further reflect on the
dynamics that drive evaluation and the role they can play in facilitating a discussion around the vi
Socio - economic empowerment in city slums, The case of Nairobi adolescent girls
Phelix Odipo, Jaboma Allan
EXPOSÉ/PAPER
In 2012, Nairobi Youth Centre implemented a girls' empowerment livelihoods project in collaboration with
International Rescue Committee with grant support from the Nike Foundation. The project goal was to promote
socio-economic empowerment of adolescent girls in Kenya through sustainable micro franchise projects. At the end
of the project assessments were carried out in the slums to determine the impact of the project on adolescent girls
and their livelihoods. A total of 125 girls were surveyed through individual questionnaires and focus group
discussions. The methodology involved qualitative and quantitative evaluation methods. Key assessment findings
were that female mentors boost girls' participation. Savings are important to girls, Girls are motivated by training
that respond to their distinct needs. Livelihoods reduce significantly the likelihood of early marriages, gender -based
violence and motherhood
29
Terminological dictionary project in evaluation: an attempt at clarity
Ghislain Arbour, Richard Marceau, Francine Sylvain
EXPOSÉ/PAPER
Terminology is the scientific study of the specialized language associated with a domain of activity, like law or
medicine. It proposes methods and criteria to develop and improve a language by supporting the selection of its
terms and the quality of its definitions. Program evaluation, as a domain of activity, requires a clear specialized
language not only to facilitate seamless communication among the specialis ts (the scholars and practitioners), but
also to deliver consistent and unambiguous evaluation products. Sources of evaluation terms and definitions are
abundant, but sometimes can generate confusion and lack consistency. This presentation explains the relevance,
methodology and the characteristics of the project of a dictionary of evaluation in English that utilizes the teachings
of terminology to address a fundamental challenge of communication in our discipline.
The basics of system evaluation: A case illustration
Ralph Renger
Systems evaluation is receiving considerable attention in the evaluation literature. In 2012 the GIZ in Germany
sponsored a conference devoted specifically to understanding the potential of syste ms evaluation. Conclusions from
the proceedings were mixed; while it was acknowledged systems evaluation has great potential there was no shared
definition of systems evaluation. Another criticism levied against systems evaluation is it is too abstract, wh ich is a
major limiting factor in bridging from theory to practice. Three years later there has been very little progress in
applied systems evaluation. This presentation will share the systems evaluation model developed by the author born
out of meeting the challenges faced in evaluating the cardiac systems of care for seven states. Robust system
evaluation principles will be shared and illustrated using examples from the cardiac system of care evaluation.
The economic value of Quebec’s Healthy Lifestyle Habits Promotion Strategy
Pierre-Alexandre Dionne, Tarik Benmarhnia, Ericc Tchouaket, Alvine Fansi, Astrid Brousselle
This study aims at evaluating the economic value of the healthy lifestyle habits promotion (HLH P) strategy in
Quebec. As effectiveness data was not available, instead of estimating total savings associated with observed
effects, we estimated the break-even point beyond which the economic benefits of the HLHP strategy would surpass
its costs. Direct healthcare expenditures associated to unhealthy habits were $4.16 billion in 2010 -11, 47% of which
were attributable to risk factors targeted by the HLHP strategy habits (smoking, inactivity, insufficient fruits and
vegetables intake, obesity). We concluded that the HLHP cost ($110 M) corresponded to only 5.6% of health
expenditures attributable to these risk factors. Using a novel methodology, this study compared the economic value
of HLHP activities in Quebec against healthcare expenditures associated wit h risk factors. This study fits within the
emerging movement of public health economic evaluation.
University equity practitioners as evaluators: barriers and facilitators to effective evaluation
Ryan Naylor
EXPOSÉ/PAPER
Student equity is a major higher education policy concern in Australia. Equity program evaluation is an essential
component of social inclusion in higher education, but evaluation is rarely built into the core business of equity
programs. It is often an afterthought, without sufficient rigo ur or detail. Practitioners are best situated to regularly
and cost-effectively evaluate programs to enhance social inclusion, but report barriers in terms of their
understanding and intrinsic and extrinsic motivations. Increased understanding of barriers and possibilities
associated with equity evaluation will significantly assist the effective operation of equity programs, creating
improved outcomes for disadvantaged students. This study will examine the processes used to evaluate equity
programs at a major Australian university, identify some barriers to effective evaluation, and describe the
development of resources to support the professional development of equity practitioners as evaluators.
30
Using SenseMaker as an evaluative tool for a girls' education intervention in Ethiopia
Rebecca Smith, Rita Fierro
EXPOSÉ/PAPER
In May 2014, Girl Hub Ethiopia worked with Dr. Rita Fierro and Jarco Consulting to conduct a research project using
the SenseMaker story collection methodology to explore barriers to educati on for pastoralist girls in Afar region,
Ethiopia.
What can Evaluation do to support Performance Measurement?
Dwane Unruh, Clinton Hynes
EXPOSÉ/PAPER
A strong Performance Measurement (PM) Strategy is essential to the identification and collection of data to support
evaluation. However, since PM Strategies are not always developed with evaluation in mind, or are not developed at
all, administrative data are frequently inadequate to the purpose of evaluation. As a result, evaluators must use
expensive surveys or other methods to make up for this limitation. This presentation will outline challenges in
developing PM Strategies, provide examples of innovative methods to deal with inadequate data, review approaches
to building stronger connections between evaluation and PM, and highlight particular cases that illustrate promising
practices. The examples provided will be drawn primarily from the presenters’ collective experience working as
evaluators in the federal government, while also relying on examples from o ne presenter's extensive experience
outside of government developing programs and designing PM Strategies.
World café - New Meaning-Making Strategies through Dialogue
Chris Corrigan, Amy Lenzo
WORLD CAFÉ/WORLD CAFÉ
8h30-8h45
Mots d'ouverture / Prise en charge du flambeau de l’Année internationale de l’évaluation
Co-Présidents
PLÉNIÈRE/PLENARY
8h45-9h15
Plénière d'ouverture: Un monde meilleur/A better world
Laure Waridel
PLÉNIÈRE/PLENARY
9h15-10h
Values and evaluation: can and how can the evaluator engage to defend values for a better world?
Ernest House, Maria Bustelo, Moderated By Astrid Broussselle
PLÉNIÈRE/PLENARY
31
26 MAI
MARDI TUESDAY
MAY
26
10h30-11h30
CES Professional Learning: Focus on Intermediate Learning
Sandra Sellick, Kathy Gerber, Jacques Bérard
Participants in this session will come away with new information about CES professional learning opportunities. This
will include information about the recently revised Essential Skills Series and CES webinars. However the focus will
be on initiatives to increase options for intermediate level learning. This will include an overview of the new
intermediate professional learning plan and its implications for course development in 2015 - 2016 as well the
options to be created for the professional learning of credentialed and uncredentialed evaluators. The round table
will also provide participants with the opportunity to ask questions, share information about professional learning
options across Canada and internationally, and provide recommendations regarding priorities for the CES
Professional Learning Committee in the year ahead. This session will be particularly valuable for Chapter Council
professional learning coordinators.
Food for Thought in Evaluating Complex, Collaborative Initiatives
Jennifer Yessis, Scott Graham, Barbara Zupko
EXPOSÉ/PAPER
Nourishing School Communities (NSC) is a collaborative, evidence-based initiative that aims to get healthy and local
foods into the minds and onto the plates of school children across Can ada. Many partners are engaged in this
project including children, teachers, principals, school meal providers, farmers, after -school providers, First Nations
communities, researchers and policy makers. Together we are supporting changes to school food env ironments,
changes that aim to shift policies and practices in a way that makes healthy local foods easier to access and fun to
eat. Collective impact of NSC is being assessed using a developmental evaluation approach. Evaluators from different
settings are supporting school communities with tools and resources to assess changes to their school and after school environments. Presenters will describe infrastructure and roles played by partners in this evaluation, and the
benefits and challenges of this approach.
Measuring Collaboration: Relationships as Results
Sarah Farina
EXPOSÉ/PAPER
Achieving lasting outcomes and impacts increasingly depends on strong and effective relationships between
organizations and individuals. Evaluating relationships as results places governance at the center of effective
implementation. This presentation looks at concepts and tools for measuring collaboration. Participants will take
away an inventory of approaches to measuring collaboration as well as a decision -support for understanding the role
of collaboration in different environments.
Missing Link between Program Evaluation and Public Policy?
Joanne Barry, Michelle Riddle
EXPOSÉ/PAPER
Program evaluation addresses whether or not programs/services are working as intended and/or are achieving their
intended outcomes. Through the examination of several large-scale evaluations of government funded programs,
this presentation will assess the impact of program evaluation on public policy. The presentation will focus on
challenges and best practice learning from these evaluations in order to assess whether program evaluation
influences or transforms public policy and therefore the lives of the end users.
32
Moving targets: Pragmatic considerations for evaluating innovative and complex interventions
Dominique Leonard, Mythri Vijendran
This discussion focuses on challenges and opportunities when evaluating complex, innovative interventions that are
in constant flux. Facilitated by 2 evaluators from SRDC, the discussion draws on lessons learned from evaluations of
2 complex interventions: a research collaborative for integrated healthcare delivery and an innovative after -school
program to improve educational outcomes for youth. Participants will share their cha llenges in similar situations,
whether due to program "drift" or complete overhaul, competing visions of the intervention, or rapid organizational
change. Facilitators will ask questions to stimulate thinking about different evaluation approaches, especial ly when
developmental evaluation is not feasible. Focus will also be on sharing coping strategies, particularly for
intermediate evaluators who may not be positioned to influence overall evaluation goals, but who can nevertheless
contribute to the success of "moving target' evaluations.
Performance Measurement and Evaluation: We need to talk.
Juliana Bravo Hernandez, Mary Kay Lamarche
The aim of this presentation is to encourage evaluation practitioners to strategize in th eir day to day work as they
deal with performance measurement and evaluation tensions to make room for flexible context -specific learning and
reflection. The presentation will argue the importance of being aware and critical of how the over emphasis on
results and evidence and the resulting discourses, is resulting in an environment where tools and methods are
leading the work rather than and not dialogue and thinking. As we strive to comply with reporting, whatever that is
in our operational contexts, we prioritize tasks and deliverables but do we equally assign importance and relevance
to learning? Where can we build and foster learning environments? We will invite a cold hard look at the work we do
and how we do it. As we see it, we need to talk.
Redonner place aux valeurs dans les processus et critères d'évaluation
Guy Cauquil
Les valeurs servent-elles encore de critère pour apprécier la valeur de l'action publique et pour choisir une démarche
d'évaluation? Peu. L'évaluation de la performance publique vise davantage la rationalisation de l'action et
l'efficience que la pertinence et l'utilité. Les valeurs éthiques ont cédé le pas à l'imaginaire économique et culturel
occidental qui privilégie la satisfaction et la compétition plutôt que la solidarité ou l'équité Faire de l'évaluation pour
un monde meilleur invite donc à interroger les valeurs qui fondent l'évaluation ainsi que leurs déterminants
sociologiques, culturels ou économiques. A quelles valeurs l'évaluatio n peut-elle faire référence pour apprécier la
valeur d'une action publique? En fonction de quels critères éthiques, économiques ou opérationnels accorde -t-on de
la valeur à telle ou telle méthode? Les citoyens ont-ils ou non leur mot à dire? Ils ont aussi leurs valeurs
The Skilful Means of Multistakeholder Evaluation
Daniel Buckles
EXPOSÉ/PAPER
The way we carry out evaluation is as critical as the learning and the transformation we seek. Conventional views on
evaluation treat the mastery of methods and tools simply as means towards ends. The skilful means of participatory
evaluation described in this presentation head in a different direction, towards the embeddedness of ends in means.
When seen from this perspective, skills, methods and technology are n ot mere instruments with instruction manuals
to be followed, as if without purpose and meaning. To support genuine accountability and learning the selection and
scaling of tools must be reconciled with what stakeholders are and long to be, that is, thought ful beings that strive
to reach out and communicate with other human beings and the world that surrounds us. More attention needs to
be given not only to the feedback loops key to evaluation but also skillful means for emergent learning across social
and cultural boundaries.
33
Understanding and Applying Innovation in Federal Government Program Evaluation
Tyler Toso, Blayne Beacham
EXPOSÉ/PAPER
The purpose of this presentation is to summarize current knowledge related to innovation in program evaluation in
order to inform future thinking and improve evaluation practice through exposure to these ideas. This project
achieves this purpose by identifying innovations occurring in program evaluation, while discussing the benefits,
drawbacks, barriers and lessons learned regarding innovation in the context of federal government program
evaluation. This presentation draws on 5 years of literature from Canadian Journal of Program Evaluation, American
Journal of Evaluation & New Directions for Evaluation in order to ident ify innovative tools and methodologies that
had demonstrated potential for use in program evaluation. In addition, 17 key informant interviews were conducted
with evaluators working in a federal government context to further identify successful innovations , and gather
expert insight on how to incorporate innovations into evaluation.
Understanding and evaluating government administrative services
Isabelle Mercier, Eugenia Didenko
EXPOSÉ/PAPER
Governments provide an array of administrative services as a means to fulfil legislative requirements and meet
operational needs. Most prominent include procurement, information technology, real property management and
others. The delivery of these services consumes a large portion of the government's budget and civil se rvice
capacity. With an increasing pressure to rationalize spending, departments are transforming their administrative
operations, with a view of a leaner, more efficient service delivery. Evaluating administrative services, however, is
becoming more complex, as services are being reformed through centralization, modernization and outsourcing. This
presentation will highlight the nature and unique elements central to public administrative services and ways to
assess their performance. The discussion will explore the centrality of user satisfaction, client relationship, service
utilization, service standards and cost savings measures in the administrative context.
10h30-11h45
#eval: Twitter as a potential line of evidence in program evaluations
Douglas Hagar
EXPOSÉ/PAPER
Social media data has been used to support marketing campaigns and academic projects exploring issue networks
and information dissemination, but to date the use of social media data in program evaluations has largely been
unexplored. This paper examines the potential to use tweets as a line of evidence in program evaluations, identifies
tools and options for data collection and analysis, and discusses practical and ethical issues pertaining to the use of
Twitter data. The findings of the paper draw upon existing academic research and interviews with government
officials and evaluators. The paper provides suggestions for those evaluators considering using tweets as a line of
evidence in program evaluations in the future.
Communicating your evaluative messages: how knowledge mobilization and technology can help
Kate Svensson, Barbara Szijarto
New technologies are opening up a world of possibilities for evaluators to avoid 'the dusty shelf' — that dreaded
place where unused findings are said to languish. This is particularly the case when technology is used in alignment
with effective knowledge mobilisation (KMb) practices. The field of KMb offers substantive guidance to facilitate
learning among individuals from different professions and perspectives — such as evaluators, policymakers and
program administrators. This paper focuses on selected KMb practices salient to sharing evaluative messages with
stakeholders, providing background on why they are considered effective. Special attention will be paid to how this
34
work may be made easier and less costly with the use of technology (social network mapping and infographics used
as examples). The paper also offers suggestions for how to build a KMb plan into your evaluation project.
Comparaison des choix méthodologiques opérés par les évaluateurs de programme novices et experts
Maud Mediell, Eric Dionne
L'évaluation de programme (EP) présente de nombreux enjeux relatifs à la praxis, enjeux qui montrent l'importance
de mieux documenter comment les évaluateurs, novices et experts, appréhendent le processus d'évaluation et, plus
particulièrement, la méthodologie. Le but de cette présentation qui s'inscrit dans une recherche doctorale est
d'aborder les pratiques des évaluateurs de programme, et plus particulièrement leurs choix méthodologiques, et la
potentielle influence des parties prenantes sur ces processus décisionnels lors de leurs EP. Ainsi présenterons -nous
notre devis de recherche et les instruments de collecte de données que nous avons construits afin de répondre à la
question principale de recherche suivante : en quoi les évaluateurs novices et les évaluateurs experts ½uvrant dans
un même domaine de pratique se distinguent-ils, ou non, dans les choix méthodologiques qu'ils mettent en ½uvre
relativement à la réalisation de leurs EP?
Dictionnaire terminologique de l'évaluation: la dimension conceptuelle
Richard Marceau, Francine Sylvain
EXPOSÉ/PAPER
Les évaluateurs ont avantage à stabiliser leur terminologie pour mieux construire leur domaine de pratique et le
faire évoluer. Le domaine de l'évaluation a atteint la maturité nécessaire à l'établissement d'une terminologie
cohérente qui le définit de manière identitaire et qui est en mesure de résoudre de manière pragmatique beaucoup
de difficultés de communication entre les évaluateurs et leurs parties prenantes et entre les évaluateurs eux -mêmes.
C'est dans ce contexte que les auteurs présentent le Dictionnaire terminologique de l'évaluation. Après un bref
retour sur la problématique, ils abordent le système conceptuel ayant servi de base au choix de la nomenclature, la
méthodologie lexicographique ainsi que la structure de chaque entrée du Dictionnaire : dé finition, exemples,
iconographie, remarques et d'autres rubriques qui permettent de saisir l'ancrage de la terminologie évaluative
proposée dans la langue française.
EVALUATION AND THE REFORM OF THE SOCIAL ASSISTANCE SYSTEM IN THE STATE OF TABASCO, MEXICO
Mónica Ruiz-Casares, David Tobis, Elsy Alcalá Cortés
EXPOSÉ/PAPER
These are challenging times for Governments around the world. Growing inequity, global interdependence, and
citizens' pressures for transparency and accountability in public administration are leading many countries to
introduce comprehensive reforms to deal more effectively with the challenges and opportunities they face. The
State of Tabasco (Mexico), rich in natural resources, still confronts widespread poverty, gender -based violence,
migration, youth suicide, and an ageing population. In 2014, the new state government commissioned a rapid
assessment of the social assistance system, with special attention to the Desarrollo Integral de la Familia (DIF), the
main social assistance agency in the state. Intensive desk review, key informant interviews, and site visits to
programs for children, women and families, people with disabilities, the elderly, and poor and marginalized
communities, were conducted. This paper will describe the process and results of this assessment a
Evaluation of ISFM technologies in Southern Highlands of Tanzania
William Mmari, Kissa Mwaisoba
EXPOSÉ/PAPER
Soil fertility degradation and striga weed in maize production system are key problems leading to low maize
productivity, addressed by the project . The project aims to address the problems through increased awareness and
knowledge of farmers on integrated soil fertility management, for increased soil productivity. The main objective
was to assess the status of use of ISFM technologies among farmers growing maize and beans in Mbeya rural and
35
Mbozi. A cross-sectional design was used, in which data was collected at a single point in time. This design is the
mostly preferred because it is cost effective and less time consuming. Both qualitative and quantitative data were
collected and analyzed. Mbeya rural and Mbozi district were purposive selected because the project "Intensifying
maize and beans production through improved soil health and marketing in Mbeya, Tanzan ia. The proportion of the
farmers using fertilize increased from 80 to 99.1% after project intervention.
Generating the tough discussions - the power of your evaluations
Marie-Anik Gagne, Shelley Borys
EXPOSÉ/PAPER
We all know that evaluations are informative, necessary and awesome — well at least the converted attending this
conference know that this is true. But seriously, your evaluation findings can generate the really tough discussions
that need to take place in government! This presentation will provide examples of the types key discussions senior
management at Health Canada and the Public Health Agency of Canada have had as a result of evaluation reports —
all while respecting committee confidences of course. These are tough discussions that C anadians want senior
bureaucrats to have on a regular basis: When is the federal role's done? How do we compare internationally to other
governments or private companies that offer the same/similar services? Are we impacting individual Canadians? Are
we using our resources most efficiently and effectively? Are we doing what we should be doing? Your evaluations
really are informative, necessary and awesome!
Graduate Evaluators, How Well Do You Do?
Shufang Dong
Understanding graduate evaluators is important to graduate schools and the field of PE, given the acknowledged and
potential contribution of graduate evaluators. What are their experiences as graduate student evaluators? What can
be learnt from their experiences? The focus of this research is to explore the experiences of graduate evaluators,
through which, at the same time, to investigate and discuss the methodological approaches in PE. This study is
conducted with qualitative methodological approaches such as document and text study, interviews through the
analytic framework of grounded theory and critical discourse analysis under socio -cultural theoretical framework. It
is hoped that the methodologies in this study can add thoughts to program evaluation metho dology in that these
methodologies can both contribute to the analysis of the data but have their own uniqueness which is derived from
the essence of PE. It is also hoped that this study can be an starting point of future study.
Le problème comme réponse : cerner les critères de mérite par l'étude des problèmes sociaux
Ghislain Arbour
EXPOSÉ/PAPER
Pour juger une intervention, nous devons comprendre les raisons qui font qu'elle est nécessaire. À cet égard, on ne
peut trouver des critères de mérite pertinents sans comprendre le problème social qui justifie (peut-être) le fait
d'intervenir. Cette présentation suggère un cadre d'analyse pour étudier les problèmes sociaux de manière à ce que
nous puissions identifier de bons critères de mérite dans les évaluation s des interventions publiques. Le cadre
aborde trois dimensions du problème social: (a) sa nature morale (pourquoi est -il mauvais?); (b) sa mécanique
causale (comment il fonctionne?); et (c) son intensité (dans quelle mesure est -on en mauvaise posture?). Le cadre
explique comment organiser ces questions de telle manière qu'ils puissent ensemble dévoiler la pertinence sociale
des programmes gouvernementaux. Le cadre sera également analysé pour son potentiel à aborder les questions
d'équité dans l'offre des programmes publics.
Learning from Evaluation Studies of Literacy Campaigns in India
Kaustubha Nand Bhatt
EXPOSÉ/PAPER
36
India, a nation with one third illiterates of the world, strives to eradicate adult illiteracy. A National Literacy Mission
(NLM) was established in 1988 to place functional literacy for all on the national agenda. Total Literacy Campaigns
(TLC) were initiated in 1989. An appropriate system of the evaluation was launched simultaneously to provide a
reliable feed-back to the concerned to facilitate and ensure effective implementation of literacy programs. G.B. Pant
Social Science Institute, Allahabad conducted three evaluation studies. These evaluation exercises aimed to study
the overall performance of TLC in the concerned district. It examined the operational strategies and implementation
approaches in the context of approved plan of action having regard to district specific factors; identified the strength
and weaknesses of the project; analyzed the factors responsible for the strength a nd weaknesses; and suggested
corrective and remedial measures.
L'utilisation des connaissances issues de l'évaluation : étude de cas organisationnelle
Isabelle Bourgeois, Clémence Naré
EXPOSÉ/PAPER
Le gouvernement fédéral a mené un exercice semblable à celui de la révision des programmes qui a lieu
actuellement au Québec entre 2007 et 2010. Les ministères devaient, à l'aide de données probantes issues de
l'évaluation, entre autres, identifier des économies de 5% et 10% de leur enveloppe budgétaire. Ce que l 'on a
constaté, suite à cet exercice, c'est que dans la majorité des cas, les ministères ne disposaient pas des données
évaluatives nécessaires pour prendre ces décisions budgétaires. Notre étude, financée par le CRSH et dont nous
présenterons les résultats préliminaires, porte sur la contribution de l'évaluation au processus décisionnel dans deux
agences fédérales depuis la fin de l'examen stratégique. Nous identifierons, entre autres, les caractéristiques des
rapports d'évaluation et les éléments-clés du contexte organisationnel qui sont susceptibles de mener à une plus
grande utilisation des données probantes par la haute gestion.
Rapid Assessment of Teacher Effectiveness (RATE): Creating the World We Want with Predictive Eval
Michael Strong, John Gargani
EXPOSÉ/PAPER
Expectations of instant information are changing the way evaluators work. In addition to retrospective summative
evaluations, we now engage in real-time formative and developmental evaluation. The next step is predictive
evaluation — quickly predicting how effective an intervention will be and immediately using that information to
improve outcomes. In this presentation, we describe our efforts to develop a predictive evaluation tool — the Rapid
Assessment of Teacher Effectiveness (RATE) — that allows us to predict the future success of teachers by observing
short video clips of their instruction. By making predictions early in the school year, administrators can support
teachers as necessary, increase student achievement, and minimize costs. We present evidence of RATE's predictive
validity and describe our first field study currently underway with pre -service and early-career teachers. The
implications of predictive evaluation for the field are discussed.
Strengthening the Utility of Competencies for Canadian Evaluation Practice
Gunter Rochow, Pierre Morin
EXPOSÉ/PAPER
When the Canadian Evaluation Society developed its Credentialing Program, it considered it essential to be able to
identify the required underlying technical, personal and ethical competencies. In the view of the presenters it is
essential to go beyond the mere identification of the competencies to a fully -fledged analysis of each competency
through a participatory analysis process of practising professionals which would help publi c, private and educational
entities in curricular planning with a view to creating a homogeneous body of knowledge, technical skills and
attitudes. The resulting profiles would also be useful in personnel recruitment, training, performance assessment
and the development of remuneration scales. The adoption of such a professional approach to competency
definition would further enhance the worldwide recognition of the Canadian Evaluation Society as a leader in
evaluation-related professional development.
37
The inner workings of a meaningful meta-evaluation
Indrani Barrón
EXPOSÉ/PAPER
The inner workings of a meaningful meta-evaluation The advancement of a culture of evaluation that is rapidly
consolidating has made some space for meta-evaluations, something that is better known as an evaluation of
evaluations. The OECD, Glossary of Key Terms in Evaluation and Results Based Management specifies that the term
is used for evaluations designed to aggregate findings from a series of evaluations. It can also be used to denote the
evaluation of an evaluation to judge its quality and/or assess the performance of the evaluators. The examples on
which our paper is based are UNICEF's Global Evaluation Quality Assurance System (GEROS), and the Meta Evaluation of Project and Programme Evaluations 2012-14 for the Ministry for Foreign Affairs of Finland. UNICEF's
GEROS is a system used to ensure that UNICEF's evaluations uphold the high quality standards set for them. The
GEROS review process and the information provided in annual
12h-12h15
Passation du flambeau 2015 « au prochain sur la liste » et la présentation des résultats de la consultation sur
l’Agenda global.
Benoit Gauthier, Stewart Donaldson, Lyn Alderman, Serge Eric Yakeu, Maria Bustello, Estaban Tapella, Guy
Cauquil, Fation Luli, Olivier Sossa
LANCEMENT/LAUNCH
13h15-14h45
An Impact Evaluation of the MCC Jordan Compact Water and Wastewater Projects: Baseline Results
Nathan Cutler
EXPOSÉ/PAPER
Social Impact (SI) has been contracted by the Millennium Challenge Corporation (MCC) to measure the impact of
water and wastewater infrastructure activities on economic and social outcomes in Jordan, which represents the
first attempt to conduct a rigorous impact evaluation (IE) of a large infrastructure project in Jordan. SI has de signed
a quasi-experimental IE aiming to establish a causal relationship between infrastructure investments and observed
changes in household availability and consumption of water and household income, expenditure and health
indicators using a multiple treatment and multiple control propensity score matching methodology. Baseline data,
including attempting to model the willingness to pay for piped water availability, are discussed alongside a
demonstration of a novel, semi-automatic Stata package used to check the quality of data from computer assisted
personal interviewing (CAPI).
Broad impact – enhancing people’s prospects: Lessons from GIZ's corporate strategy evaluation
Frank Dubert
EXPOSÉ/PAPER
As a federal enterprise, GIZ supports the German Government in achieving its objectives in the field of international
cooperation for sustainable development. The evaluation office of GIZ recently completed a corporate -level
evaluation on scaling up in GIZ-supported development programs. With its work, GIZ aims to improve people’s living
conditions and give them new prospects. The corporate strategy evaluation was devoted to the question of how, and
how successfully, scaling up is achieved in the projects implemented by GIZ and its partners. The evaluation
comprised two interconnected parts, namely a portfolio review of more than 300 projects, and seven country case
studies in the water sector. Spaces, drivers and pathways that that can be used to implement scaling up processes
and that partially determine the success of scaling up mechanisms impact were analyzed. As a learning organization,
GIZ uses the evaluation findings for a company-wide learning process.
38
Evaluation of Collective Impact Initiatives: Two Contrasting Examples
Nancy Carter, Robert Chatwin
EXPOSÉ/PAPER
The presentation will discuss the application of the collective impact framework to two initiatives in Nova Scotia and
the evaluation frameworks that have been developed to assess each initiative. The CIHR Reform Assessment of
Impact Group (CRAIG) - a Maritime initiative aiming to evaluate impacts of restructuring of health research funding
in the region - has adopted a collective impact approach and is using developmental and participatory approaches to
evaluation for the purpose of learning from the experience. The Regulated health Professions Network (RHPN) was
formed as a result of provincial legislation that encourages collaboration amongst stakeholders to address common
issues. The RHPN has designed an evaluation framework aimed at accountability and demonstrating value of this
innovative approach to regulation. Again a developmental and participatory approach have been taken to ensure
engagement and learning amongst members. Key aspects of Collective Impact will be discussed.
Evolving Developmental Evaluation: Challenges & solutions from 4 perspectives
Kate Woodman , Eugene Krupa, Cheryl Poth, Tara Hanson
PANEL/PANEL
Developmental Evaluation (DE) theory provides a model to address many challenges faced by evaluators of programs
in dynamic contexts. However, when applied in real-life with all the complexities people introduce, many challenges
emerge for those responsible for the evaluation, and considerable "in -progress re-thinking" is always needed.
Although many of the themes are consistent among projects, challenges can also vary depending on the sector,
project focus, context, project duration, people involved and their backgrounds, capacities and personalities. In this
interactive panel session, we examine the challenges from 4 players' p erspectives: evaluator on the ground (Krupa),
evaluation project manager (Woodman), funder's contract manager (Hanson) and DE researcher (Poth). We pose
paths to resolution from our experiences, and suggest implications and recommendations for professional
competencies, project management, contract management and future research.
Expérience de suivi des indicateurs communs sur l'eau potable en zone rurale au Burkina Faso
Lynda Rey, Pierre-Yves Charpentier
EXPOSÉ/PAPER
Le Projet Eau de la Haute Comoé au Burkina-Faso (PEHC) vise la transformation des conditions de vie de la
population de cinq communes à travers notamment un accès durable à l'eau potable. Dans ce projet, le suivi évaluation a constitué un défi important pour les évaluateurs. Il a fallu trouver un équilibre entre les indicateurs
standards développés au niveau international d'une part, et d'autre part contextualiser ces indicateurs pour mettre
en valeur les réalités locales influençant la performance du projet. Nous présentons ici les défis rencontrés et leçons
apprises du système de suivi-évaluation des indicateurs liés à l'eau potable du PEHC (milieu d'intervention rural et
isolé, coûts importants associés à la collecte et à l'analyse des donnée, etc.). En conclusion, la contextualisation des
indicateurs standards de performance constitue «un art» qui relève à la fois de la compétence professionnelle et de
la sensibilité culturelle de l'évaluateur.
Growth and Performance of Africa VOPEs: The case of The Evaluation Society of Kenya (ESK)
Jaboma Allan, Julius Nyangaga
EXPOSÉ/PAPER
ESK is a network of M&E practitioners in Kenya and the eastern Africa region and was initiated in 2008. The main
objective of the Society is to promote a sustainable and quality monitoring and evaluation practice for improving
development policies, Projects and programs. The society delivers professional input into Kenya's development
agenda through multistakeholder collaborations. Between January and June 2013, the Society undert ook a study to
develop a usable database of M & E services and products in the country and identify skills gaps that will inform a
resource mobilization and capacity building strategy. The survey targeted individuals and institutions providing M&E
services in the country. Gaps identified included data analysis, designing and implementing of evaluations among
39
others. The study reveals that despite a large presence of M&E professionals, more innovative ways have to be
found to enhance recruitment, membership and sharing of M&E products for development.
Jeunes Évaluateurs: Comment faire sa place dans le monde de l'évaluation?
Jérôme Gandin, Miché Ouedraogo, Andréanne Martel, Seyive Wilfried Affodegon, Marie-Pier Marchand,
Marie Gervais
PANEL/PANEL
Ce panel questionne la place des Évaluateurs Émergents dans le monde de l’évaluation. Il propose aux participants
de discuter quatre questions avec des intervenants aux profils, cultures et perspectives variés:1)Que devrait faire un
évaluateur émergent pour tendre à devenir un bon évaluateur ?2)Quels sont les principaux défis que vous avez
rencontrés(ou rencontrez actuellement) pour développer votre pratique en évaluation? Quelles sont les stratégies
utilisées pour y faire face?3)Comme évaluateur émergent, quelle est vo tre expérience en lien avec votre société
d’évaluation? Qu’est-ce qui pourrait être fait pour renforcer la place et la contribution des évaluateurs émergents
dans votre société?4)Des démarches sont actuellement en cours pour mettre en place un réseau inter national
d'évaluateurs émergents francophones. Quels devraient être les objectifs visés par un tel réseau? Quels en seraient
les facteurs de réussite? Votre contribution potentielle?
Made in Africa Evaluation: an overview of innovative capacity development programs
Mirianaud Oswald Agbadome
EXPOSÉ/PAPER
Monitoring and Evaluation of Rule of Law Initiatives
Sadie Yang, Andrea Redway, Colleen Duggan, Joel Martinez
PANEL/PANEL
The rule of law (RoL) is widely recognized as an important development goal. However effective monitoring and
evaluation of RoL programs is difficult: there is no commonly accepted definition of RoL; attribution for RoL results is
often unclear; and the selection of indicators and collection of data to demonstrate results often proves to b e
challenging. The panel will contrast approaches that best demonstrate whether RoL results have been achieved at
the sectoral, project and individual intervention level: • WJP will share indices to measure the degree to which
national justice systems as a whole have been strengthened by RoL initiatives; • The CBA will compare distinct
evaluation approaches that have been applied to its projects and their relative effectiveness in measuring a project’s
impact on RoL; • IDRC will contrast methodologies used to evaluate solutions to RoL challenges and the extent to
which they generate evidence on the equitable delivery of justice services.
Outcomes Evaluation of Multi Sector Partnerships for Sustainable Community Development in Nigeria
Anthony Chovwen
EXPOSÉ/PAPER
The General Memorandum of Understanding(GMoU) as a development interface strategy is widely used by
multinationals and international oil companies working in the oil rich Niger delta of Nigeria in numerous hosts
communities as a multi sector partnership for sustainable community development. This study assess the impact of
Gentoil GMoU Outcomes in selected post 5 years tenured communities in 3 states of the Niger Delta, Delta, Rivers
and Bayelsa. Preliminary results shows the strategy has had varying d egree of effectiveness across the participating
communities. Key performance indicators (KPI) such as transparency and accountability, capacity building,
community empowerment, participations and inclusiveness, business climate, governance and democracy ha ve been
achieved in participating communities. Progress towards sustainability, partnership with development agencies,
interface with other agencies are remain low points of the strategy.
Partenariat et évaluation au service de l’innovation sociale dans le milieu des Premières Nations
Ana Gherghel, Caroline St-Louis, Patricia Montambault , Nadine Rousselot
40
PANEL/PANEL
Comment une évaluation collaborative de type formatif peut-elle contribuer au renforcement de la gouvernance
d’'une organisation des Premières Nations? Comment s’assurer que les résultats d’'évaluation soient utilisés comme
outils de gestion? Quelles stratégies peuvent favoriser l’'appropriation des résultats à la fois par les milieux de
pratique, par les gestionnaires et par les communautés? Les présentatrices réfléchissent conjointement à ces
questions sur la base d’'une expérience récente de partenariat stratégique établi entre la Commission de santé et
services sociaux des Premières Nations du Québec et du Labrador (CSSSPNQL) et Avenir d’'en fants. Croiser les points
de vue des gestionnaires et des évaluateurs permet de voir sous différents angles les étapes franchies, les défis
relevés, les conditions favorables, les leçons apprises et les modalités d ’échange développées pour faciliter
l’appropriation des savoirs issus de l’'évaluation.
Placing the Evaluation Practice in the heart of policy making, A case of Uganda
Juliet Carolyn Anewa Odeke
EXPOSÉ/PAPER
In Uganda, there is a broad consensus that evaluations and the practice is crucial in innovating development
solutions and shaping policy. This paper will attempt to highlight which ways evaluation is placed in the heart of
policy making in Uganda. The various institutional and legal frameworks present, would be roles, challenges, what
has worked versus what has. Contextual issues will be raised to facilitate dialog in the Conference "For the world we
want" Achieving influence at national level requires: Highlight success stories (Gov't& CSOs) Democratic debates
(parliament) Capacity: basic trainings for policy makers (ECD) Space for inexperienced evaluators to learn Mentoring
at national level Access and Sharing: the media, dissemination seminars Active associations e.g. Uganda Evaluation
Association A central body of knowledge e.g. GEF Exposure to best practices Online & offline debates Making clear
evaluation mandates to enhance ownership.
Roles of evaluation in enhancing Nigerian Government climate change policies and programmes
Onyinyechi Ogbonna, Juliana Iwuchukwu
EXPOSÉ/PAPER
This paper reviewed government Climate change (CC) policies and programmes in Nigeria and possible roles of
evaluation. Nigeria government has reactions and moves toward tackling CC problems by way of invention of policies
and programme. Among them are: in 2010, the National Assembly passed a bill to create a national Climate Change
commission to facilitate coordination and support for the multi-level and cross-sectoral adaptation responses. Also,
the Government of Nigeria embarked upon the development of National Ad aptation Strategy and Plan of Action on
Climate Change for Nigeria. However, many policies and programmes are hampered by numerous challenges. The
paper therefore recommends possible ways of involving evaluators for proper coordination of their activities so as to
spur synergies among them in order to actually mitigate and adapt to CC which is their ultimate goal and enhance
the efforts of government in this aspect.
Selection challenges in evaluation design in Northern Nigeria
Bukola Oyinloye
EXPOSÉ/PAPER
This presentation succinctly summarises the challenges encountered during the data collection for the baseline
impact and outcome evaluation for a teacher training programme in Northern Nigeria. To promote local ownership,
the evaluation sample selection criteria were shared with the programme's national stakeholders who used it to
select the required elements for the evaluation. However, the data collection process revealed a number of issues
with the selected elements, as a large proportion did not fulfill the provided criteria. To address these issues, sample
restoration mechanisms were employed in the field during the remaining data collection period. Ultimately, the
process resulted in a number of lessons learned both for the programme, the evaluation s ervice provider, and the
programme's national stakeholders and such lessons will inform subsequent sample selection processes for the
programme.
41
The Use of Developmental Evaluation in Organizational Practice
Robert Mclean, Joanna Kocsis, Chaitali Sinha, John Gargani
PANEL/PANEL
Striving to improve quality and impact are pressing considerations for most organizations. How can this be achieved
in dynamic, complex settings? We are living in a world that demands concrete results, yet recognizes the ‘messy’
way change occurs. How can this be reconciled? This panel focuses on one particular organization’s experience
exploring, implementing and using an approach that is designed to respond to such a conundrum. Developmental
Evaluation (DE) is an approach to real-time learning designed to evaluate efforts unfolding in uncertain, dynamic
contexts. The International Development Research Centre is learning from past experiences applying DE and
adopting a learning-by-doing approach moving forward. Panellists will provide an overview of using DE at IDRC,
share findings from an IDRC research study on how DE is used in different North American grant -making
organizations, a case study of applying DE in a program at IDRC, and commentary on DE from an expert perspe ctive.
Trends in humanitarian evaluation: insights from the pilot of ALNAP guidance
Alexandra Warner
EXPOSÉ/PAPER
For evaluation to stay relevant in the fluid and fast paced environments of humanitarian action, conventional
evaluation methods and approaches need to be adapted and contextualised. In 2010, ALNAP started developing the
first comprehensive, utilisation-focused and user-oriented guide on Evaluating Humanitarian Action (EHA),
www.alnap.org/eha. The 18-month, active pilot process of the guide closed at the end of 2014. The pilot gave ALNAP
insights into tacit aspects of EHA processes and identified trends in practice. Some of which may hamper the role
and influence evaluation can have on improving programming and fostering learning. In this presentation, an ALNAP
researcher will share some of these trends, interesting lessons learned, practical examples of how organisations
have improved EHA processes, as well as some approaches and tools highlighted through the pilot process that could
aid in furthering the influence of evaluations in this sector.
United Nations Evaluation Group (UNEG) Resource Pack on Joint Evaluation
Krishna Belbase, Scott Greene
EXPOSÉ/PAPER
There is growing demand for joint evaluations that are credible and useful both in terms of ensuring accountability
for results and for lesson learning. The United Nations Evaluation Group (UNEG) has recently issued a Resource Pack
on Joint Evaluation which consists of two components: a) A Joint Evaluation Guidance Document which outlines
main issues that arise when conducting a joint evaluation; b) A Toolkit, which includes a collection of documents,
including examples of good practices and UNEG-specific guidance on planning, managing and utilizing joint
evaluations. The co-chairs of the UNEG Taskforce would like to share the process and content of the joint evaluation
resource pack with the CES community. The presentation will focus on the overall participatory approach adopted in
formulating the resource pack, quality assurance process, the actual content of the guidance document and the
toolkit and their dissemination.
What Did We Learn?: Experiences Implementing Theory-Based Approaches in the Federal Government
Context
Jane Whynot, Steve Montague, Mary Kay Lamarche, Eric Seraphim
PANEL/PANEL
Since the 2012 issuance of Theory-Based Approaches to Evaluation: Concepts and Practices, federal government
evaluators have been absorbed in comprehensively unpacking how an intervention is intended to work, in order to
demonstrate achieved results. Various theory-based approaches have been undertaken by different departments
and sectors across government, targeting various intervention levels. Panelists identify articulated, and commonly
experienced challenges in working with these approaches, responding with both the solutions implemented. These
challenges include resourcing levels, diverse responsibilities, participatory methodologies, and aligning informational
42
content with the core issues associated with government evaluation. Panelists will offer ins ight on plausible
solutions to ensure that decision-makers have access to the quality and timely information offered by evaluation.
What is and is not essential for a quality impact evaluation?
Reuben Ford, Susanna Gurr, David Gyarmati, Jean-Pierre Voyer
PANEL/PANEL
Some program impact evaluations are attempted in circumstances where they are not appropriate or cannot hope
deliver a valid or reliable answer. Other evaluations avoid impact estimation when it would be possible and even
advisable for the future of the program. An experienced panel of evaluators from SRDC and the BC Centre for
Employment Excellence will speak to what they consider essential (and not) for a quality impact evaluation to take
place. Topics start with the setting of the evaluation question, the state of development of the program and its
implementation. Panelists will review means to control credibly for participant selection, design (including ethical)
issues, prospects for measuring the right outcomes, partnerships and commitm ents to rigour and trust. They will
seek to dispel myths that randomized trials are the only option, that impact evaluations are necessarily expensive
and require high program fidelity and extensive data to achieve their goals.
13h15-15h15
Evaluation Capacity-building for Leaders: A Mobile Learning Tool
Kylie Hutchinson, Chris Lovato
Capacity-building in evaluation has tended to focus on program-level coordinators and staff, ignoring the key role
that leaders play in using evaluation to make better decisions. To fully achieve the potential evaluation has in
making informed decisions and using practice-based evidence depends, to a large extent, on leaders and decisionmakers who are better informed consumers of evaluation. How can we best maximize the use of evaluation in
planning and decision-making through capacity building with these busy individuals? We have developed a free
mobile-learning package that provides this target audience key information about evaluation and how to best work
with evaluators. A beta version of this unique approach to capacity building with leaders and decision makers will be
demonstrated in this session.
14h45-16h15
Animal Evaluation; Why should the evaluator talk with the cow?
David Brous
EXPOSÉ/PAPER
The evaluation of programs that target animals as their significant stakeholders present special challenges for
evaluators. How do we establish appropriate channels for communicating with animals, to ensure that the animal's
perspective on the impact of programs is gained, such as those concerned with the animal welfare standards for the
care of livestock, standards for transportation and ethical processes for slaughtering? Recent policy development on
the transfer of live sheep and cattle to international markets from Australia has highlighted the importance of
establishing the animal's perspective in evaluating the outcomes of policies to enhance animal welfare. Access to
international markets for livestock produce requires consideration of animal needs. This paper will explore the
issues of conducting evaluations in animal centred contexts and draw on the presenter's experience in assessment of
biosecurity and livestock market access programs in Australia
Assessment of implementation fidelity of the Arctic Char Distribution Program in Nunavik
Lara Gautier, Catherine Pirkle, Christopher Furgal, Michel Lucas
43
In September 2011, the Nunavik Regional Board of Health and Social Se rvices began the Arctic Char Distribution
Program (AC-DP) for Pregnant Women. This program promotes the consumption of Arctic Char — a preferred
traditional Inuit food that is nutritionally rich and relatively contaminant -free — by pregnant women living in villages
along the Hudson Bay, in Nunavik. This intervention intends to reduce exposure to contaminants and improve food
security in Inuit communities. However, implementation of the program is incomplete since it does not currently
cover all intended areas, and its environmental and financial sustainability are at stake. We assess its
implementation based on data collected from program documentation, meeting minutes, field notes, and qualitative
interviews with program recipients and implementers. Themes emerging from a review and coding of these
materials are discussed in light of the framework for implementation fidelity developed by Carroll et al, 2007.
Bâtir une culture d'évaluation dans un établissement de services en contexte de diversité
Gabrielle Lemieux, Monica Ruiz-Casares
EXPOSÉ/PAPER
Comme ailleurs au Canada, la population du territoire du CSSS de la Montagne est caractérisée par la pluriethnicité
et par plusieurs facteurs de vulnérabilité qui l’accompagnent. La recherche, l’adaptation et l’innovation permettent
depuis plusieurs années aux organisations de mieux répondre aux besoins des usagers de diverses origines. Depuis
plus récemment, l’évaluation se présente comme un moyen pour tous de réfléchir, de remettre en question leurs
pratiques et d’apprendre pour améliorer encore plus la qualité des services et ainsi réduire les inégalités. Cette
culture d’évaluation émergeante doit être soutenue par les organisations afin de développer les capacités
d’évaluation et optimiser l’utilisation des connaissances. Les établissements de services peuvent s’inspirer de
l’expérience du CSSS dans l’élaboration d’une structure et d’une politique d’évaluation par une approche
participative, grâce à laquelle ces outils sont devenus de puissants promoteurs de la culture évaluative.
Canadian Evaluation support of International Evaluation-Challenges, Achievements & Lessons
Robert Lahey, Charles Lusthaus, Jean-Serge Quesnel, Laila Smith, Boubacar Aw
PANEL/PANEL
With 2015 the international Year of Evaluation, it is useful to examine the role Canada has played to support
Evaluation capacity development around the globe. Where the keynote speaker to the 2014 CES conference noted
“Evaluation needs more Canada”, the session will highlight the myriad of ways Canadian Eva luation has indeed
supported and influenced the practice of Evaluation around the world. Canada can also learn from this international
experience, and so the panel will also reflect on ‘lessons’ for Evaluation in Canada to be gleaned from this
international exposure. Finally, the panel will look beyond 2015 and provide commentary on priority areas for
Evaluation where we ought to focus our energies, both in Canada and globally, following the 'Year of Evaluation'.
Each panelist has a unique relationship with Evaluation in Canada and significant experience in supporting and
conducting Evaluation in a wide range of countries around the globe.
Comparative Economic Analysis of Beekeeping using Traditional and Improved Beehives in Tanzania
Nicholaus Kuboja
The overall objective of the study was to undertake a comparative economic analysis of beekeeping using either
improved or traditional beehives. Data was collected from 69 beekeepers that were purposively selecte d in the
study area. Budgetary and profitability ratio analysis were carried -on. Both beekeeping using improved and
traditional beehives was profitable. However, profitability between the two differs; beekeepers using traditional
beehives realized a net farm income of TZS 1034485.00 against TZS 351713.70 which was realized from using
improved beehives. Also a return on investment per 1TZS recorded using traditional beehives was 4.267 against
1.524 accrued from improved beehives
44
Défis et opportunités pour l'évaluation à travers le monde - points de vue de présidents d'associations /
Challenges and opportunities for evaluation around the world - views from association presidents
CES - Benoit Gauthier, AEA - Stewart Donaldson, AES - Lyn Alderman, AFREA - Serge Eric Yakeu, EES - Maria
Bustello, RELAC - Estaban Tapella, RFE - Guy Cauquil, Saep- Fation Luli, SQEP - Olivier Sossa
PLÉNIÈRE/PLENARY
Evaluating human rights mainstreaming - different from equity-focused evaluation?
Sabine Becker-Thierry
Development and human rights have for a long time been seen as contradicting paradigms, yet the Right to
Development was declared (1986) and a 'Human Rights-Based Approach to planning' (HRBA; 2003) been more and
more followed. Yet, not all organizations and projects explicitly apply a HRBA, rather they 'mainstream' human rights
(in addition to achieving a specific development goal, they are meant to also positively contribute to human rights).
The discussion on ways of assessing HRBA and human-rights mainstreaming is not explicitly building on development
evaluation approaches, whether gender evaluation, feminist evaluation, transformative/ empowerment evaluation
and the recent equity-focus evaluation. Is an evaluation with an equity-focus synonymous to evaluating human
rights in a development intervention? What does evaluating human rights mainstreaming in a development
intervention mean? And what are possible entry points?
Evaluating program impacts: does the source of outcome data matter?
Reuben Ford
EXPOSÉ/PAPER
Using multiple data sources offers evaluators many advantages including expanding the set of observations,
contrasting results and validating interpretations. Also, multi-year evaluations often see the range of data sources
change over time. Future to Discover is a randomized study that estimates impacts of education interventions over
time while transitioning from using a combination of administrative datasets and follow -up surveys to a combination
of administrative datasets only. The study assesses the consequences of switching between data sources on
substantive findings and describes four factors that affect the robustness of program impact estimates: the coverage
of outcomes, survey error, data matching practices, and conventions for the treatment of missing observations.
These factors, individually or combined with each other, can affect the validity and reliability of the resulting
program impact estimates.
Evaluation and nuclear waste management in France: actor repertoires, learning and mistrust
Markku Lehtonen
EXPOSÉ/PAPER
After a long, conflict-ridden history, the French plans for deep geological disposal of medium and high -level
radioactive waste have reached a stage at which the repository to be built between 2018 and 2025. This complex
and contested "megaproject" promises to benefit its economically declining host region, yet it also generates
significant health, environmental, and socio-economic concerns. Drawing on fieldwork in the planned host region,
this paper examines the potential and the pitfalls of evaluation of the socio -economic aspects in the governance of
the project, in a conflict-ridden context characterised by widespread mistrust between actors, and resistance and
scepticism towards evaluation. This context of mistrust, together with the deeply entrenched positions of several
key actors, tends to hinder the capacity of evaluation to foster learning. Yet, the criticism and mistrust against the
project could also be harnessed for creating conditions favourable to learning.
Evaluations that make a difference: What we have learned from around the world
Burt Perrin, Rochelle Zorzi, Martha McGuire
PANEL/PANEL
45
The project Evaluations that Make a Difference: Stories from around the world (Evaluation Stories) received an
EvalPartners Innovation Challenge Award to promote the use of evaluation. Evaluation Stories is using the
universally accessible form of stories to share examples of how evaluations have made a true difference to the lives
of program recipients. Ten stories have been chosen from six regions (Europe, Africa, Australasia, South
America/Caribbean, Asia and North America). In this panel, we will share some of the stories of evaluations that
have led to changes in people's lives. The stories are told from the perspective of evaluation users and program
participants, and provide colourful examples of how evaluation can be a force for social betterment. The panellists
will draw lessons from across the stories about how to undertake evaluation s o that it is more likely to result in
positive change.
From First International Evaluation Conference to UN Year of Evaluation - What has Changed?
Andy Rowe, Linda Lee, Shelley Borys, John Mayne, Francois Dumaine
PANEL/PANEL
The panel includes four CES Fellows whose work spans the period from before the First International Evaluation
Conference co-sponsored by CES in 1995 to 2015 and this International Year of Evaluation and will be chaired by a
former CES President. They will identify important changes in evaluation over that period and present their views on
some of these. The audience will be asked to identify additional important changes and from these select some for
further discussion by the audience and panel. A summary from the chair will use the dis cussions to identify
important gains in evaluation and point to important issues that evaluation should address.
Gender-responsive program evaluation: Perspectives from UN agencies, public sector, and consultants
Cyuma Mbayiha, Daniel Sansfaçon, Emmanuel Trépanier, Jane Whynot, Vanessa Anastasopoulos
PANEL/PANEL
Recognition of the importance of incorporating gender into evaluation policies and practices has been growing
steadily across national and international evaluation communities. This presentation bri ngs together a diverse group
gender and evaluation specialists from UN Women, Universalia, the University of Ottawa, and Status of Women
Canada, to explore the role of gender in evaluation. Panelists will share their unique perspectives regarding the
value and objectives of gender-responsive evaluation, how it is carried out within their work and organizations,
lessons learned, and challenges faced. Through this forum, approaches and methodological considerations aimed at
inspiring the implementation of gender-responsive evaluation will be presented.
Improvement of development outcomes through an evaluation: Uganda’s evaluation of its Poverty Eradic
Albert Byamugisha, Alex Bashasha Turyatemba
EXPOSÉ/PAPER
The evaluation of Uganda’s Poverty Eradication Action Plan (PEAP) is a rare example of a nationally-driven evaluation
of a country’s poverty reduction strategy whose findings have potentially made an improvement in the livelihoods of
the people by: influencing policy and improvement in the performance of Gov ernment programs. Uganda’s PEAP
started in 1997, and was the first national poverty plan in Africa, pre -dating and informing the World Banksupported Poverty Reduction Strategy Papers that spread across the continent thereafter. The PEAP was updated in
2000/01 and in 2003/04. By 2007, the Government decided a new direction and new type of plan was needed, and
set about designing a broad ranging evaluation that would provide a measure of what had been achieved under
PEAP, and importantly set the direction for the new plan. This proved insightful, with cross-government coordination
of the evaluation leading to evaluation findings discussed by Cabinet under a white paper, and lessons
Introduction des Smartphones dans la collecte quantitative pour l'évaluation
Isabelle Agier, Valéry Ridde
EXPOSÉ/PAPER
La rapidité de mise à disposition de données probantes est un atout essentiel pour l'évaluation des interventions.
Les enquêtes (papier) ont vu leur efficacité s'améliorer avec l'apparition des PDA (fin 80) et des téléphones
46
intelligents (2000). Cela a constitué des gains successifs d'efficacité et de coûts pour l'évaluation et imposé de
repenser le travail de collecte à toutes les étapes. Cette communication montre comment le téléphone intelligent a
permis l'amélioration de la collecte de données par rapport au papier et au PDA. Se basant sur l'expérience de sept
collectes en 2012 et 2013 dans le cadre d'un programme de recherche au Burkina Faso (IRSC), nous décrivons les
défis technologiques rencontrés sur le terrain, les réponses apportés et les adaptations nécessaires tout au long de
la démarche. Enfin, nous exposons les nouvelles opportunités offertes pour améliorer la rapidité de l'accessibilité
des données.
L'analyse logique comme outil de changement des pratiques favorisant l'accès aux soins
Marie Beauséjour, Mylaine Breton, Michael Eshiemokhai, Jean Ouellet, Astrid Brousselle
L'analyse logique permet de porter un jugement sur le bien-fondé d’une intervention en examinant sa nature et en
remettant en cause ses modalités. Nous y avons ici recours pour soutenir l'élaboration d'une intervention de triage
raisonné des demandes de consultation et de priorisation des patients en clinique de soins spécialisés. Ces stratégies
sont réputées efficaces pour favoriser un accès approprié et en temps opportun aux soins de santé. Or, il s'agit
surtout d'initiatives locales et peu d'auteurs ont tenté de modéliser ces approches pour en comprendre les
composantes clés. Ce travail a consisté à comparer le modèle logique d'une intervention proposée par les
professionnels d'un hôpital montréalais à un cadre de référence élaboré d'une revue de la littérature afin de
proposer la meilleure intervention possible pour le milieu. L'intérêt de cett e approche en soutien à la prise de
décision sera discuté.
L'analyse textuelle par ordinateur : un outil utile à l'évaluation de la mise en oeuvre ?
Jean Bélanger, Gilles Roy, François Royer
EXPOSÉ/PAPER
L'évaluation de la mise en oeuvre d'une intervention nécessite d'en documenter les stades initiaux de mobilisation
et d'appropriation. L'évaluation de ces étapes nécessite généralement le recours à des méthodes de type
qualitatives qui nécessite un temps d'analyse assez long, limitant ainsi un retour rapide aux acteurs pour qu'ils
puissent utiliser cette information. L'appel à des méthodes d'analyse assistée par ordinateur est possiblement une
piste prometteuse afin de lever ce problème. La présente conférence vise donc à vérifier, à partir de données
provenant de deux évaluations de mise en oeuvre, l'utilité réelle de logiciels tels que ALCESTE et SÉMATO en
comparant les résultats d'analyse produits sur des réponses de plus de 700 répondants, à ceux générés à partir
d'une analyse thématique manuelle. Les résultats montrent une utilisation prometteuse de ce type d'outil tout en
présentant ses limites.
Measuring and building evaluation capacity in Ontario public health units
Isabelle Bourgeois, Louise Simmons, Nikolas Hotte, Raïmi Osseni
PANEL/PANEL
Ontario public health units (PHU) are responsible for developing and implementing various public health programs in
their communities. Each unit is expected to conduct program evaluations when new interventions are implemented
or when there is evidence of unexpected operational issues or program results. Increasing accountability pressures
and evolving performance management systems aimed at demonstrating the value of its programs and services are
driving PHU to enhance their local program evaluation efforts. This panel will present various aspects of a
collaborative study conducted to measure the evaluation capacity of 28 PHU. The objective of the study was to
identify key organizational characteristics associated with higher levels of evaluation capacity, in order to support
future capacity development. Data were collected through the use of a standardized tool developed by Bourgeois et
al (2013), as well as key informant interviews with a representative from each of the PHU.
Mixing methods for strengthening impact evaluations
Shagun Sabarwal, Heather Lanthorn
47
EXPOSÉ/PAPER
Multiple forms of data and analysis enable us to not only make causal claims about what works in development but
also why, when and where which assumptions (in a theory of change) are met. We pres ent guidelines for conducting
high-quality and theory-driven impact evaluations (IE) built on a meaningful mix of methodologies. By mixed methods (MM), we specifically refer to integrating qualitative and quantitative research methods for data collection,
analysis and interpretation. The present paper will produce a pre-analysis plan template for conducting qualitative
research as a part of an IE and specific guidelines for conducting high quality MM IEs that fulfill the goal of learning
what works, why, and in what contexts. We will review all MM IEs currently included in 3ie's IE Repository, which is
an index of all published IEs of development interventions. We address an important lacuna: moving from an
expressed goal to engage in more MM IEs to providing a blueprint of how to do so
Monitoring Impact Funds
Leonardo Santos, Maria Parreiras, Renato Gouvêa
EXPOSÉ/PAPER
Investment Impact Funds are similar to Venture Capital funds in several aspects, but with one fundamental
difference; their performance is also measured by the benefits provided to lower income classes. Funds invested in
startups induce management and governance changes, suggest projects to raise capacity and search for new
business opportunities As a contribution to the impact funds management, the developed model presented assists
the monitoring from investor's internal activities to the observation of social benefits and return of investment. Two
models used by the development banks to plan, monitor, evaluate and communicate their investment performance
were used to create the final model to monitor investment impact funds. Investment impact funds complexity was
represented by a synthesis developed by Roduner and Ambrose of two models: Logical Framework and “Outcome
Mapping.
Multidimensional poverty index approach for program evaluation, an experience from Afghanistan
Abdullah Al Mamun
EXPOSÉ/PAPER
In many countries, the policymakers of donor and partner agencies consider the results of multiple indicators to
measure the status of poverty reduction improvement programs. They also use the results of aforesaid indicators to
decide on further investment to the programs. For example, financing to those partner agencies show positive
results of a set of indicators. Sometimes, agencies are only interested to consider households' income as an indicator
to measure the status a program. This paper will argue that although many donor and partner agencies consider to
use the results of multiple indicators at a time or use of a single income related indicato r such as 'households'
income' to measure the status, such as success or failure of a poverty reduction, it is better to consider an additional
single index that consists of the results of concerned indicators using 'Multidimensional Poverty Index (MPI)'.
Partners in learning through evaluative thinking to transform and build the capacity to use evidence
Keiko Kuji-Shikatani, Doris Mcwhorter, Joannie Leung, Dianne Oliphant, Mary Richards
PANEL/PANEL
Our experience over the past decade illustrates the power of evaluation to improve and transform education
systems. Guided by the ministry’s Research and Evaluation Strategy (2005), evaluation and evaluative thinking have
been built into policy and program design and implementation. Partnerships between internal and external
evaluators mutually build evaluation capacity, contributing to increasing the positive outcomes for all Ontario’s
students. This presentation will illustrate the achievements and challenges of evaluation and its potential to guide
and inform the implementation of Ontario’s Renewed Vision for Education (2014) including, a five -phase, mixedmethods evaluation on the impact, spread, adaptability and sustainability of Collaborative Inquiry as a professional
learning approach, and evaluation of initiative that offers differentiated and intense supports to district school
boards to build principals capacity as the instructional leader for student success and ECB efforts.
48
Sustainable Equity Programs in Higher Education: Designing and Testing an Evaluation Resource
Jenny De Vries
Evaluation of equity programs is a vital aspect of the Australian higher education system, but equity practitioners
feel unsupported in this task. This presentation details the development of a resource to assist in this task. During
2014, through a competitive grants scheme, a total of $181M, was distributed by the Australian Federal Government
so that programs could be delivered across Outreach, Access and Support programs. Ong oing funding is not assured
and universities must make hard decisions on which programs are having the most impact in this equity space. This
case study is situated in one of the eight Western Australian universities which all compete for funding. This cas e
study research will develop and trial an evaluation resource to assist practitioners to determine if their many and
varied programs are actually making a difference to the people identified within the various equity groups. A mixed
method research design has been selected for this study.
Utiliser les téléphones mobiles pour évaluer l'impact des projets agricoles
Samuel Kouakou
EXPOSÉ/PAPER
A l'instar de nombreux pays africains, la Côte d'Ivoire fait face à une forte augmentation de l'utilisation des
téléphones mobiles en milieu rural, particulièrement par les agriculteurs. Cette évaluation pilote a été réalisée, en
testant les possibilités de mettre en ½uvre un système par SMS, pour évaluer l'impact socio -économique de l'Exprojet de riziculture financé par l'Union européenne dans la région Centre-Nord. FrontlineSMS est un logiciel libre
qui établit un double sens messagerie-texte en utilisant seulement un ordinateur, un modem et des téléphones
cellulaires GSM. Mais, existe-t-il suffisamment de connaissances au sein des évaluateurs pour utiliser efficacement
cette technologie? il ressort de l'étude que cette technologie innovante offre une opportunité, de réduire la
distance, le temps et le coût de l'évaluation des projets. Cependant, il reste nécessaire d'a méliorer la connectivité en
milieu rural pour une mise en ½uvre réussie.
Launch of CES Annual Conference Fellows' Strand
Andy Rowe, Linda Lee, Shelley Borys, John Mayne, Francois Dumaine
LANCEMENT/LAUNCH
14h45-16h35
Comment identifier les processus performants d'une intervention ergonomique ?
Valérie Albert, Henriette Bilodeau, Nicole Vézina, Fabien Coutarel
PRÉSENTATION AFFICHE/POSTER PRESENTATION
Une intervention ergonomique visant à améliorer la santé des travailleurs et la production est conçue sur mesure et
peut comporter plus d'une centaine d'actions (processus) influencées par le contexte de l'entreprise. Alors que les
modèles d'évaluation dominants en ergonomie ne s'intéressent qu'aux effets produits, il semble que l'aspect
généralisable de l'intervention concerne plutôt les processus «performants», soit ceux qui ont généré les effets dans
un contexte particulier. Une étude-pilote a donc permis de dégager des indicateurs à inclure dans un nouveau
modèle d'évaluation des interventions ergonomiques. Celui-ci devra se baser sur une analyse quantitative et
qualitative des processus, inclure les éléments-clés du contexte qui ont influencé les processus et intégrer les
perspectives de l'ergonome et d'acteurs-clés de l'entreprise pour documenter les mécanismes de production des
effets.
Diffusion de Ces années incroyables: Évaluation de son implantation en protection de l'enfance
Isabelle-Ann Leclair M.
49
CAI, PEHP probant largement diffusé, est implanté depuis 2003 au Centre jeunesse de Montréal (CJM). Le modèle
d'action de Chen stipule que 6 éléments interagissent dans l'implantation de programmes (organisation; personnes
qui implantent; partenaires; contexte écologique; protocole d'intervention/livraison de servi ce; population cible). La
fidélité d'implantation résulte de l'adéquation entre ce qui est prévu dans ce modèle et ce qui est fait en réalité.
Cette présentation décrit à partir de ce modèle, l'implantation de CAI au CJM. Seront présentés les résultats
provenant de données colligées depuis plus de 10 ans (plus de 34 groupes, 20 animateurs, 248 participants)
concernant dosage, participation, qualité et adhérence au programme. Les résultats suggèrent que l'implantation est
fidèle à la planification. L'accent sera mis sur le processus d'évaluation continu et la collaboration entre le milieu
clinique et de la recherche.
EVALUABILITY OF A STRATEGY FOR INTEGRATING EDUCATION-SERVICE-COMMUNITY IN HEALTH AREA
Marly Marques, Catia Oliveira, Raquel Torres, Ana Cristina Reis, Solange Kanso, Aline Leal, Silvia Carvalho
The Program for the Enhancement of the Primary Care Professional (PROVAB) is an initiative by the Brazilian Ministry
of Health established in 2011 and designed to address one of the challenges of the National Health Service: the lack
of healthcare professionals on the outskirts of big cities, the countryside and remote areas. This study aimed to
conduct an Evaluability Assessment (EA) of PROVAB that stages: identifi cation of stakeholders, strategic analysis,
modeling of the intervention and recommendations. Data collection: review of official documents, meetings and
semi-structured interviews. The Results showed that PROVAB will be able to contribute in reducing ineq ualities in
access to health care; however, the study revealed some weaknesses in management that need to be reviewed for
the second cycle of the program, such as increased dialogue with the programs doctors, more investment in
educational technologies and institutional support and improvement of the information flow.
Evaluating the Evidence on the Efficiency of Performance-Based Financing in Lower Income Countries
Anne-Marie Turcotte-Tremblay, Jessica Spagnolo, Manuela De Allegri, Valéry Ridde
Context: Performance-based financing (PBF) is being implemented in low- and middle- income countries (LMICs) to
improve health care services and health outcomes. The objective of this literature review is to evaluate the streng th
of the evidence regarding the efficiency of PBF in LMICs. Methods: First, we scanned the reference lists of two
systematic reviews on the efficiency of PBF to identify articles that met the established inclusion/exclusion criteria.
Subsequently, we searched for new English and French articles in PubMed, EconLit, Google Scholar and Google.
Results & Discussion: Seven articles were included in this review, and classified as partial economic evaluations.
None of the included studies made a clear connection between the costs of PBF in LMICs and its effects. Based on
these results, we reject the proposition that PBF initiatives are empirically proven to be efficient interventions in
health care delivery in LMICs.
Evaluation as a Reflective Practice
Nathan Raja
Reflection questions, W5H1, shorter form: what, so what, and now what
Evaluation du Programme Micro FEM (SGP) au Sénégal
Ndeye Fatou Diop, Dieynaba Dia
Le Programme de Micro-Financement du Fonds pour l'Environnement Mondial (PMF/FEM) est financé par le FEM
pour appuyer et promouvoir les initiatives d'actions communautaires visant à faire face aux grands problèmes
d'environnement dans le monde. Ceci à travers la mise en ½uvre de projets (d'un montant maximal d'USD 50 000),
relatifs à cinq domaines de priorité du FEM : la diversité biologique, le changement climatique, les eaux
internationales, la gestion durable des terres et les polluants organiques persistants. Lancé en 1993 au Sénégal, le
50
PMF/FEM est exécuté par le PNUD pays en collaboration avec l'Etat du Sénégal. Les bénéficiaires sont les ONG et les
OCB .Depuis sa création, le PMF a subventionné plus de 260 projets. L'évaluation conjointe FEM/PNUD lancée au
mois de mai-juin 2014 entre dans le cadre de la préparation du sixième plan opérationnel et de l'évaluation du
programme global qui a concerné 8 pays.
Evaluation of the Supervision Needs and Efficiency of an NCR Agency
Julie Gosselin, Sophie-Claire Valiquette-Tessier, Marie-Pier Vandette
In many helping professions, clinical supervision is the main vehicle for professional development (Falender &
Shafranske, 2007). Thus, the purpose of this evaluation, request ed by a National capital region (NCR) agency, was to
identify the needs, challenges, and strengths of supervisors and supervisees, and assess the efficacy of current
supervision practices. The lines of evidence consisted of a document and file review, grou p discussions, and
interviews. First, findings indicated a need for additional concrete resources, and a need to develop a more
competency-based supervision organizational culture. Second, respondents noted a number of strengths
contributing to the efficacy of current supervision practices, including the agency's commitment to its core values.
Third, there was recognition of supervision's impact on crisis management, but less so on more regular work
activities. Finally, respondents identified ways in which supervision activities could be enhanced.
'Experts' learning from communities: Lessons from rural East Africa
Njoroge Kamau
Majority of development agencies strive to fully involve target beneficiaries in project mo nitoring. Communities may
participate in initial project start up phases but not monitoring and eventual evaluation often. Community Led Total
Sanitation (CLTS) is a sanitation and hygiene promotion strategy that aims at inspiring communities to stop open
defecation. Unlike traditional sanitation and hygiene approaches, CLTS advocates for non -subsidy and encourages
communities to use locally available construction materials to build latrines. The community decides that all
community members will build and use latrines. Monitoring of the progress towards eradication of open defecation
is also done by the community. During this project, we were pleasantly surprised to see that the target communities
designed their own monitoring tools. Even more surprising is that the tool developed by community members was
richer and more informative than our 'expert' monitoring tool.
Focus Groups - Misconceptions, Implications and Insights
Atena Bishka
Focus groups - misconceptions, implications and insights
Leçons apprises : Évaluations et méthodes pour enquêter auprès des populations difficiles à joindre
Jérémie Butoyi, Valéry Ridde, Susanna Ogunnaike-Cooke
Cette affiche présente les leçons apprises à travers différents projets d'études - évaluation de faisabilité portant sur
la santé des populations difficiles à joindre (sans papiers, PPVIH) en déhors des structures de services connues. Elle
montre comment créer les conditions de mise en oeuvre et d'acceptabilité des projets et présente solutions
innovatrices pour optimiser l'accès à une populations diversifiées en particulier celles qui mènent une vie
clandestine. Elle aborde aussi les méthodes d'opérationnalisation du recrutement à travers un processus d'éducation
et de responsabilisation qui transforme le participant en complice pour une action rapide en misant sur la nécessité
de se prendre en main. L'évaluateur peut rapidement accéder aux usagers potentiels utilisant la méthode du temps
et de l'espace (TLS)pour avoir des données de première mains.
51
M&E Translated: Tools for Participatory & Sustainable Systems for International Implementers
Elizabeth Lewis
Developing sustainable monitoring and evaluation (M&E) systems is a challenge for any implementing organization;
international organizations face additional cultural and logistical challenges. The International Republican Institute
(IRI) — a non-partisan, non-profit organization committed to advancing democracy worldwide — has worked since
2008 to develop M&E systems that can be translated across language, culture and organizational capacity. IRI has
adapted best practices from the evaluation community to be applied to programs designed to deve lop democratic
institutions, including political parties, legislative or executive government and civil society. This poster will highlight
three tools IRI developed in its quest to promote an organizational culture that values M&E, including an M&E
capacity building training curriculum for use with local staff and partners, a handbook for M&E of democracy
assistance programs, and finally an M&E systems diagnostic tool.
Models for providing gluten-free meals in hospitals & continuing care facilities: A cost-analysis
Nour Hammami
Individuals with celiac disease world-wide (~43% of Canadians) have difficulty obtaining gluten-free (GF) meals,
including in hospitals and retirement homes. Immediate attention is needed sinc e celiac patients admitted to
hospitals and continuing care facilities (CCF) require proper nourishment to maintain good health. The objective of
this study was (1) to identify the most cost efficient strategy to provide GF meals for celiac disease patie nts in a
hospital in Lebanon and (2) to develop a decision tree to guide hospitals and CCF in identifying the most suited plan
for their needs in adopting GF meals. After revision of GF food chain guidelines, 4 options were compared (e.g.
purchasing new equipment for GF products, finding a dedicated contaminant free space, purchasing from a GF
bakery..) for implementation in a hospital in Lebanon. From here a decision tree was developed that will assist
facilities all over the world in providing GF meals to their patients in need of such a diet.
Panel sur les Jeunes évaluateurs africains
Aminata Diop, Amos Menard, Ousséni Kinda, Ya Cor Ndione
Ce panel de jeunes évaluateurs africains porte un regard original sur 4 sujets d’'actualité africaine: la place des
jeunes dans les politiques publiques; l’'emploi des jeunes; les jeunes évaluateurs et la professionnalisation ; la
protection des enfants et adolescents. L’'évaluation peut aider à changer les perceptions, parfoi s négatives, de la
jeunesse en valorisant leurs capacités d’'apporter des réponses innovatrices aux problèmes les concernant. Pour
cela, une rupture s’impose dans la conception, la mise en œuvre et le S&E des programmes de jeunes. Une pratique
évaluative plus souple et inclusive va établir la confiance et le respect mutuel afin de prendre au sérieux la voix des
jeunes. Il faut des méthodes adaptées au contexte africain et sensible aux valeurs, aux conditions de vie et modes de
communication des jeunes et des enfants, qui deviennent ainsi des acteurs à part entier. Comment apporter ces
changements? Quel sera le rôle des jeunes évaluateurs?
Patient partnership approach to diabetes care in Eeyou Istchee: Identifying barriers and facilitators
Elena Kuzmina, Catherine Godin, Paul Linton , Marie-Pascale Pomey
The Cree regional health care is facing many challenges for implementing a patient partnership approach to diabetes
care. The goal of this project is to identify barriers and facilitators for diabetes management as perceived by Cree
patients at the personal, organizational and environmental levels. Individual, structured interviews were conducted
with nine Cree patients registered in the Cree Diabetes Information S ystem. Our results suggest that diabetes care
could be improved by increasing Health Care Professionals' capacity to deliver culturally sensitive care, facilitating
shared decision making for diabetes management, involving patients in the quality improveme nt process,
implementing a patient-friendly follow-up system, and reinforcing a supportive environment in the community.
52
Incorporating patients' experience in the delivery of health care is a critical element for building patient partnership
care and empowering the community in order to achieve sustainable results.
Pratique de l'évaluation au Cameroun et performance des Politiques publiques
Julien Ntouda
Depuis plus d'une décennie, l'évaluation des politiques publiques est au c½ur des préoccupations des pays d'Afrique
subsaharienne. Au Cameroun, les statistiques mettent en évidence des niveaux particulièrement faible de
l'évaluation des programmes et projets publics. Le question de recherche de cet article est de savoir comment la
pratique de l'évaluation dans l'administration publique camerounaise parvient -elle à influencer la performance des
programmes/projets publics? Ainsi, son objectif est de montrer l'influence de l'évaluation sur la performance des
programmes et projets publiques camerounais. L'étude s'appuie sur les données issues de l' enquête sur la
performance de la chaine de résultat des administrations au Cameroun. Deux méthodes d'analyse seront utilisées
dans cet article. L'analyse descriptive et l'analyse économétrique. grâce à la méthode paramétrique de frontière de
production stochastique
Prevention of social violence: an integral and articulated experience, Mexico
David Guzman Matadamas
In order to contribute to the development and considering that every action sums in a positive manner the issue
describe in the presentation “Prevention of social violence: an integral and articulated experience, Mexico” the
Crime and Violence Prevention Program is been implemented true a multilevel engagement and promotion the CVPP
has accomplish the operation of accurate interventions in a sub -national intervention. As part of the international
policy of USAID, the beneficiaries had help to pilot interventions models focus in environmental design, at risk youth,
community healing, and resilience and a local communication campaign. It’s been tested that modeling theories in
practical interventions the CVPP has a second phase for extend its life and contri bute to enhance the quality of
people affected by a huge social problem: social violence.
Saying 'No:' The Challenges of Ethical Evaluation Practice using Social Work Values
Megan Elyse Williams, Laura Sundstrom
As evaluators and social workers, ethical practice is paramount. However, clients often present challenging ethical
dilemmas. Client needs take precedence, but sometimes evaluators have to say no or set boundaries. This poster will
use case examples of four ethical dilemmas commonly faced by evaluators and how they were addressed to preserve
a good client relationship, yet following the ethics set forth by the social work profession in the context of
evaluation practice.
Strengthening evaluation influence with experiential learning: lessons from University of Waterloo
Jennifer Yessis, Lisa Stockton, Ellen Maceachen
High quality relevant evaluation studies are needed to strengthen public health and health system actio ns. But
knowledge, skills and organizational cultures that enable strong evaluations are limited across Canada. Academic
participation in evaluations is especially limited. The University of Waterloo's School of Public Health and Health
Systems and the Propel Centre for Population Health Impact are initial partners in a collaborative approach to
improve health and health systems through vibrant academic and practice partnerships. Initial priorities are
graduate student practicum placements and a 1-week learning institute, both of which emphasize experiential
learning, collaboration, and mutual exchange for those conducting, supervising and using evaluations. This
presentation will describe the design of experiential learning opportunities, and early lessons such as considerations
for academic and on-site mentoring, experiences from field placements, and roles of all involved.
53
The Art of Self-Reflection Crossing Flow and EvaluationLive!
Natalie De Sole, Melanie Hwalek
Can reflective practice and self-reflection turn around a "ho hum" evaluation experience? When looking for ways to
improve an experience, self-reflection is often overlooked. However, Schön (1983), Patton (2011), and Barrington
(2012) all uphold the human element as an essential factor for success and informational uptake. This roundtable
will highlight the importance of self-reflection and provide tools for enhancing evaluation practice using self reflection, including Csikszentmihalyi's Flow model and Melanie Hwalek's EvaluationLive! practice model. Audience
members will be guided through a reflective activity from the EvaluationLive! model. Feedback from this activity will
help further the field's focus on ways to incorporate and use reflective practice t o continually self-improve our
evaluation practice.
The M&E System of the Palestinian Ministry of Education and Higher Education
Mohammad Mousa, Rabiha Ehlian
Executive Summary Since 2009, the annual M&E report focused on monitoring the EDSP Key Performance Indicators
(KPIs). As of 2012, MoE has begun to institutionalize an annual evaluation mindset and function as a key prerequisite
for the implementation of the Program Based Approach (PBA) and Sector W ide Approach (SWAP), which are
supported by the Joint Financing Arrangement (JFA). The findings for 2012 are presented cumulatively from the
2009 Baseline together with the methodology applied in the case of each indicator. Sources of data for this repo rt
are multiple. Apart from the EMIS statistics, data originated from different MoE DGs (e.g.: Assessment and
Evaluation, Field Follow Up, Counseling, CDTP, Planning, Health, Finance, Curricula etc.). Each of the eight quality
KPIs has a separate technical team, which is composed of members from DGs and university experts.
16h35-17h35
A different perspective- Can Social Return on Investment help to tell a better story?
Alison Jette, Chantal Langevin
Evaluations of federal programs need to assess both economy and efficiency. This can be a challenge for social
programs, as the impacts of the program funded by one sector can be felt in other sectors. This also makes the
collection of financial data challenging. A standardized approach to monetizing the social value created by the
program would provide the evaluation with fuller information on the benefit of the investment. Social return on
investment is one method to do this. Health Canada will be evaluating the National Native Alcoho l and Drug Abuse
Program and piloting the use of SROI with a treatment centre. With this roundtable, we will be sharing our insight
with others who have used, or are considering using, SROI as well as those engaged in the evaluation of substance
abuse treatment. We hope this group discussion will help to highlight, for those participating, the strengths and
weaknesses of SROI in this context, as well as how it can be used to influence senior management.
Applying a multi-level implementation framework to program evaluation: a case study
Eunice Chong, Adrienne Alayli-Goebbels, Anne Philipneri , Helen Cerigo, Lori Webel-Edgar, Sarah Muir,
Heather Manson
EXPOSÉ/PAPER
Implementation science is the study of methods to bridge the gap between research and practice, and aims to
understand how evidence-based programs can be effectively implemented in real-life settings. The goals of
implementation science often overlap with the goals of program evaluation. There are many existing
implementation frameworks that were developed in an attempt to understand the process of program
implementation (i.e., barriers and facilitators), and these frameworks can guide the development of evaluation plans
and questions. For the process evaluation of Ontario's Healthy Babies Healthy Ch ildren (HBHC) program, a multi-
54
level implementation framework was used to guide the development of the evaluation questions and to identify
relevant factors in the HBHC program implementation. In this presentation, the author will provide an overview of
the application of this framework to evaluation planning, present selected findings, and discuss the limitations of the
framework.
Contraintes liées l'évaluation au Cameroun : à la recherche des facteurs explicatif
Julien Ntouda
Depuis plus d'une décennie, la pratique de l'évaluation des politiques publiques est au c½ur des débats des
communauté scientifiques et professionnelles en particulier Afrique subsaharienne en particulier. Au Cameroun,
cependant la pratique de l'évaluation dans les administrations publiques reste très faible. Le questionnement à la
base de cet article et d'identifier les facteurs qui font obstacles à la pratique de l'évaluation au Cameroun. Ainsi, son
objectif est d'identifier et valoriser les facteurs qui favorise la pratique de l'évaluation au Cameroun. l'étude s'appuie
sur les données issues de l'enquêtes sur la performance de la chaine de valeurs des administrations publiques
camerounaises. Deux méthodes d'analyses seront utilisées: descriptive et économétrique.la variable dépendante
étant qualitative et dichotomique on utilisera le modèle logit.
Enhancing evaluation impact and use at Environment Canada
Nicole Michaud, Susan Wharton
EXPOSÉ/PAPER
Environment Canada's Evaluation Division has implemented a number of tools and approaches over the years aimed
at improving and monitoring the impact and use of its evaluation products and services and supporting a culture of
continuous improvement. These processes occur at each phase of the evaluation, namely at the planning,
conducting, reporting and project closure. The presentation will touch briefly on the suite of tools and processes, but
will focus primarily on new processes implemented at evaluation project closure. Building on a recent study and
evaluation project pilots conducted in other federal agencies, the Evaluation Division is currently piloting new tools
to increase stakeholder engagement at the project closure phase for enhanced evaluation use and continued
learning within the department. The presentation will outline early results from the pilots, expected and actual
benefits and lessons learned gathered through the reflections of evaluators and evaluation users.
Evaluating Programs to Inform Systems Level Change
Laura Sundstrom, Megan Elyse Williams
EXPOSÉ/PAPER
Programs are operating in systems (i.e. criminal justice, education, behavioral health, etc.), yet most evaluations are
conducted at the program level. Planning evaluations at the systems level can benefit clients in a number of ways:
the use of shared knowledge, methods, tools, data metrics, and relationship building across organizations and
programs. Through systems level thinking, utilization of evaluation results for system level change is maximized.
Presenters will describe how systems level thinking, drawn from a social work framework, informed four program
evaluations across the criminal justice system.
Evaluation for a better world: Influencing use in non favorable contexts
Astrid Brousselle, Damien Contandriopoulos
Context is the major determinant of evaluation use. Contrary to prevailing beliefs in our field, it is neither the
evaluation approach itself nor the role of the evaluator that is the main determinant of use. In fact, there are
contexts in which results are unlikely to be used, whatever the personal qualities of the evaluator and the intensity
of participation. When an evaluator is stuck in a context where it is unlikely that th e results will be used, what are
some avenues that might be pursued to increase the evaluation's potential for influence? First, based on the
framework developed in our previous work, we will describe the characteristics of non -favourable contexts as
55
compared to favourable and political contexts. Second, we will explore avenues for influencing contexts'
characteristics based on theories of the policy process (Sabatier and Weible, 2014). Third, we will identify some
ethical tensions this reading of contexts implies for evaluators' practice.
Evaluation Standards: Use and Potential to Build A Better World
Brenda Stead
The Program Evaluation Standards, Third Edition (2010), developed by the Joint Committee on Standards for
Educational Evaluation (JCSEE), has been approved by the American National Standards Institute, and adopted by
the Canadian Evaluation Society (CES). The Classroom Standards are in the final stage of revision. This session will
begin with a short presentation on the status of both these sets of Standards, and then focus on the Program
Evaluation Standards - a pillar of the CES Professional Designation Program and evaluation professionalization.
Focused discussion questions will be used to guide participants in sharing their interest in evaluation standards,
experience in or desire to applying the Standards (through work or studies) in their real world situations. Together
we will explore the challenges and opportunities in building knowledge about and capacity in the evaluation
standards of utility, feasibility, propriety, accuracy and accountability - for a better World.
Evaluations for Good Governance- Need for Enabling Environment
Rashmi Agrawal
EXPOSÉ/PAPER
Evaluations and good governance are mutually supportive and complement each other in catalyzing development.
Effectiveness of evaluations in contributing to better governance, however, is often constrained by country's socio economic and political milieu and is conditioned by factors such as lack of owner ship of evaluations, political
compulsions that prevent policy modifications, inadequate attention to use of results, conceptual and
methodological constraints in impact evaluations, evaluation quality etc. This point is brought out through examples
from Indian experience in evaluations of development interventions. The need for appropriate enabling environment
with National Evaluation policy that promotes an evaluation culture and systematizes M&E in the country is
emphasized. Capacity building and sensitization at various levels of operations is crucial to get over the constraints.
‘Community' as a major stakeholder in development and evaluations can be the agent of change.
Facilitating the Use of Evaluation Results
Derek Wilson, Chris Lovato, Tamiza Abji
EXPOSÉ/PAPER
For a learning organization, the evaluation of programs is a continuous process which does not end when a final
report is delivered. In fact, the active use of evaluation results for planning and decision -making is one of the most
important and challenging elements of the program improvement cycle. At the University of British Columbia’s
Faculty of Medicine, an evaluation and improvement process has been established whereby evaluation
recommendations are routinely brought forward to leadership for action. The monitoring elements of this process
have recently been strengthened through a series of structural and procedural enhancements. This presentation will
outline the strategies developed and how they are enabling and reinforcing use of evaluat ion results, including
positive gains, challenges, and our insights on lessons learned.
Finding a Pathway: Supporting the Development of an Evaluation Culture at the Canada Council
Linda Lee, Shannon Peet
EXPOSÉ/PAPER
An evaluation of the Canada Council's Suite of Aboriginal programs is underway, representing the first
comprehensive program evaluation at the Council. An initial step was to develop a logic model which would
represent the suite of 15 Aboriginal Arts programs. This paper presents the organizat ional context, process for
development, final logic model (now known as the Pathway) and its impact on the institution. The development
56
process needed to include input and buy-in from Council staff, as well as culminate with the creation of a 'logic
model' that would be appropriate for the Aboriginal programs. The Pathway combines the circular with the linear,
representing a coming together (or hybrid) of Indigenous (circular, holistic) and 'Western' (sequential, causal) ways
of conceptualizing the world. As important as the product was, the role the process played in supporting the
development of an evaluation culture at the Canada Council, was equally significant.
From "translation" to "transcriation": M&E for change
Elizabeth Moreira Dos Santos, Egléubia Oliveira, Gisela Cardoso
EXPOSÉ/PAPER
Recent experiences in Brazil, aiming at improving the quality of NHSS, has examined the evaluation models
addressing the performance of 'supporters' essential to the functioning of the strategic and operational netwo rks of
the Brazilian health care system. This essay seeks to explore critical topics for the development of theories that
describe the intervention, modeling the evaluation and the use of its findings. The empirical basis of this discussion
is how to evaluate the "supporter", a constructor of connections, key implementer of QualiSUS -Network', which is a
MoH strategy, in partnership with the World Bank, to promote the quality of Health Care Networks, in 15 Brazilian
regions. The theoretical approach converges the importance of the reflective pedagogical function from translation
of interests to 'transcriation' and 'transvaluation'.
Governance strengthening in the Amazon region: how to overcome this challenge?
Angela Casanova, Marly Cruz, Ligia Giovanella, Glaydes Alves, Egleubia Oliveira, Rafaela Souza, Leonardo
Cunha
The Brazilian QualiSUS-Network Project (QNP) aims to support the implementation of Health Care Networks in 15
health care regions. One of its components is governance reinforcement - the coordination of health care system by
different actors at loco-regional level. This evaluation aims to answer at what extent the QNP contributed to the
strengthening of Brazilians' National Health System (SUS) regional governance in 3 regions within the Amazon area,
Brazil. Evaluability study involved interventions modeling, discussion and agreement among stakeholders. Data
collection comprissed interviews with key actors (n=62) and documentary research (n=43). Data is being analysed
according to governance components regarding both contextual and logical analysis. Initial results show that the
intervention strengthened the governance and integration between the actors in the regions differently, conditioned
by steering mode, consensus on political priorities and influence of strategic actors.
Harmonized evaluations for scholarship and fellowship programs
Shannon Clark Larkin, Ken Stephenson
EXPOSÉ/PAPER
Increasingly, evaluation is being called upon to broaden its scop e beyond the program (e.g., to a sector level or to a
higher level of a department’s PAA). At the same time, evaluations often struggle to find a comparison group for
their program with available and usable data. Harmonizing the evaluations of multiple, si milar programs is one
solution to both of these challenges. Driven by efforts to harmonize program delivery among the Federal granting
agencies, the NSERC-SSHRC Evaluation Division developed an innovative approach to harmonize evaluations of its
scholarship and fellowship funding opportunities, aiming to tell a more fulsome story about the financing of
graduate studies in Canada. The presenters will speak about their experiences and the challenges encountered in
harmonizing evaluation designs, ensuring consistency across large-scale surveys, analyzing across large datasets, and
synthesizing high-level findings into actionable recommendations.
Innovation in large, complex performance measurement data collection using FluidReview
Kelly Mcdonald, Kara Hayne
EXPOSÉ/PAPER
57
In 2014, the Public Health Agency of Canada streamlined performance measurement data collection tools and
processes for their community based children's programs including the Community Action Program for Children
(CAPC), the Canada Prenatal Nutrition Program (CPNP) and the Aboriginal Head Start in Urban and Northern
Communities (AHSUNC). The objective was to reduce the reporting burden placed on over 700 community projects
and to develop a cost-effective process for performance measurement data collection. The Public Health Agency
developed a user friendly web based tool using FluidReview. The process of streamlining program specific tools and
adapting to an online platform was complex, complete with challenges along the way. The end result howev er was
an excellent response rate of 95%. The results are currently being analyzed and will offer national and regional
summaries to inform program evaluation and support senior management decision making.
Institutionalization of evaluation in Albania, from the bottom upwards
Fation Luli
EXPOSÉ/PAPER
Albania has already created a favorable context for the development of a culture of good governance, managing for
results and accountability. However, the evaluative function and its degree of institutionalizat ion remains
underdeveloped. The evaluation community in Albania faced to the challenge: what strategy to choose for the
institutionalization of evaluation in Albania. Should we go directly to the central government or would be better to
start with the peripheries? In our view, this was first a political process, and to a lesser extent a technical process.
According to a long-term vision, the interest for evaluation shown by Albanian municipalities concluded that,
embarked from the municipal sector, the bottom-up theoretical model could be the starting point for developing
and institutionalizing of evalua
Institutionnalisation de l’évaluation au Bénin : Bilan et perspectives
Emmanuel David-Gnahou
EXPOSÉ/PAPER
La planification participative comme outil d'évaluation en situation post-conflit en RDC
Serge Eric Yakeu Djiam
EXPOSÉ/PAPER
Après la guerre civile de 2004, les habitants sont revenus à leurs communautés respectives dans la province du
Maniema, en RDC. Le Programme d'Action Communautaire pour le Développement Durable a contribué au
renforcement de la coopération entre les organisations de la société civile, la population et l'administration locale et
soutenir leurs capacités à faire face aux besoins vitaux. L'approche de partenariat a été adoptée comme un élément
de stratégie. A travers quatre composantes tels que Agriculture, la réhabilitation des infrastructures, épargne et de
crédit, et la gouvernance locale, les activités sont intégrées au sein d'une approche participative de la communauté,
et prend en charge les besoins socio-économiques des bénéficiaires depuis 2008. La planification participative a été
développée pour évaluer les résultats du programme en termes de gouvernance, renforcer l'appropriation par les
bénéficiaires, et d'influencer les décisions communautaires.
Les mécanismes de diffusion de l'évaluation auprès des bénéficiaires de programmes fédéraux
Étienne Thériault
La communication efficace des résultats de l'évaluation représente le premier pas vers son utilisation par les
bénéficiaires; il revient donc à l'évaluateur de créer ces liens par l'entremise du rapport d'évaluation, entre autres .
Suite à cette première communication officielle, les parties prenantes externes pourraient en principe se servir des
informations présentées dans les rapports à diverses fins, dans le but d'influencer la formulation de politiques
publiques. Ce projet a donc pour but d'explorer davantage cette problématique en s'intéressant à la communication
des résultats de l'évaluation aux bénéficiaires de programmes fédéraux canadiens en répondant à la question
58
suivante: dans quelle mesure est-ce que les bénéficiaires de programmes externes connaissent et se servent des
résultats provenant des rapports d'évaluation? Cette présentation vise à présenter les résultats de recherche.
Let’s talk about evaluators’ efficiency!
Simon Roy, Janice Remai
EXPOSÉ/PAPER
In this context of fiscal constraints, evaluators are asked to assess the effectiveness and efficiency of government
programs. But what about our own evaluation activities? Based on their private sector experience, the presenters
will share their views about how evaluations can be run efficiently, including strategies to keep resources and
activities focussed through careful planning and expectations management, effective junior resource management,
and techniques to conduct fieldwork and produce reports efficiently. The presenters will argue that these
techniques can keep the costs of evaluations at a reasonable level without significantly impacting quality.
L'utilisation progressive de l'évaluation par une ONG humanitaire au Burkina Faso
Léna D'ostie-Racine, Christian Dagenais
Bien que l'évaluation dans le secteur humanitaire soit de plus en plus valorisée, les connaissances empiriques sur
l'utilisation de l'évaluation (UÉ) dans ce contexte sont limitées. Les quelques écrits scientifiques sur le sujet
décrivent une utilisation inconstante et documentent peu les déterminants de l'UÉ. L'étude qualitative a procédé par
une collecte de données en 2009 (n=15) et une deuxième en 2011 (n=17) afin d'examiner l'évoluti on de l'UÉ et les
conditions qui la détermine, dans le contexte d'une organisation d'aide humanitaire au Burkina Faso. Cette
organisation a développé une stratégie d'évaluation afin d'examiner l'implantation et les effets de son programme
d'exemption de paiement des soins de santé. Les résultats montrent l'évolution sur deux ans de l'UÉ et clarifient
certaines conditions d'UÉ particulières au secteur humanitaire.
New approaches to int'l development evaluation: what 2 Canadians learned by asking the wrong Qs
Lindsay Renaud
EXPOSÉ/PAPER
Increasingly, middle-income countries like China, India and Brazil are launching their own international development
initiatives. ‘South-South cooperation for development', as it is commonly known, is distinct from Western models of
development assistance. Commissioned by UNDP China, 2 Canadian evaluators set out with 2 research questions:
If/how are middle-income country governments evaluating their South-South cooperation? —and— Do their
approaches differ from those of Western countries (i.e. OECD-Development Assistance Committee members)? As it
turns out, some middle-income countries are embarking on more fundamentally different evaluation approaches
than the researchers' Western-assumptions-based methodology was designed to capture. In addition to sharing their
study findings, in this presentation the researchers will share their 'behind the scenes' stories of efforts and failures
to wipe the cultural tint off their researcher glasses.
Online Guide for Commissioners of Evaluations
Zoe Boutilier
EXPOSÉ/PAPER
In larger organizations like Canada's International Development Research Centre (IDRC), evaluation is often
decentralized and evaluations are commonly commissioned by program managers whose expertise is not evaluation.
These commissioners play a critical role in defining, scoping, and enabling an evaluation. It is therefore vital to
provide accessible and user-friendly guidance to these individuals as they move through the stages of commissioning
/ managing an evaluation. IDRC is partnering with BetterEval to develop an online interactive tool to guide its staff
and grantees when they find themselves responsible for an evaluation. The interactive tool deconstructs the
evaluation management process into nine major stages and provides decision -making advice and resources for each
59
stage. By bolstering the skills of the evaluation commissioner/manager, this tool should ultimately facilitate the job
of the evaluator and contribute to higher quality evaluations.
Séance de consultation sur l’Agenda global 2016-2020 /Consultation on the Global Agenda 2016-2020
Marie Gervais, Larry Bremner, Benoit Gauthier
PANEL/PANEL
The Role of Evaluation Theory in Connecting Evaluation Capacity and Culture
Kaireen Chaytor, Nancy Carter
EXPOSÉ/PAPER
Boyle & Lemaire (1999) discussed integrating evaluation into organizations using a supply and demand model.
Chaytor and Carter (2009) presented Boyle's model as a similar capacity and culture framework. Cousins'
presentation of 'do and use' could also be presented using the model. Survey data gathered by Chaytor and Carter
shows evaluators' perception of capacity strength exceeds their rating of a culture of evaluation in their own
workplace. Evaluation capacity is necessary, but not sufficient for a strong culture. In situations where evaluation
capacity is strong but culture is lacking, there is a risk of capacity being diminished or lost over ti me if culture is not
intentionally addressed. This presentation will discuss how evaluation theory can be used to integrate capacity with
culture. Literature from evaluation, organizational theory and organizational behaviour will be brought to the paper.
They Said What? Successfully Managing Multiple Languages in Evaluation
Nansy Jean-Baptiste, Lindsay Renaud
EXPOSÉ/PAPER
Evaluators are increasingly working in bilingual or multilingual environments. Examples of projects where multiple
languages may be used include work including official language minorities, Aboriginal groups, persons with hearing
or visual impairments, new Canadians, etc. Evaluators can also work on multilingual evaluations through projects
with/in other countries. This presentation will outline three main approaches to supporting full participation in
multilingual evaluations: (1) bilingual/multilingual evaluators, (2) interpreters/translators and (3) asking participants
to communicate in their non-native language. The requirements, advantages and disadvantages of each approach
will be presented. A decision-making tool will also be outlined. Throughout the presentation concrete examples of
what to do and not to do will be discussed.
What’s in a name? Defining your organization’s ‘Value for Money’
Elisabetta Micaro
EXPOSÉ/PAPER
Value for Money (VfM) is becoming the new credo of aid development. Yet the international development
community is still struggling with proposed definitions of VfM. While donors are putting pressure on organizations to
show how they deliver VfM, no comprehensive definition of this concept is yet available. In the past two years, many
attempts have been made by organizations to develop frameworks and methodologies to define and measure VfM.
Universalia has recently supported an international organization in the development of a conceptual framework for
VfM and of its operationalisation. The presentation has the objective of advancing the thinking on VfM. This will be
done in two ways; by presenting the model and the lessons learned from this experience so as to obtain feedback
from the participants on ways to improve the model; and by promoting, through a series of questions, exchanges
among participants on their understanding of this concept and sharing of their exper iences.
8h30-8h45
Lancement du Réseau francophone des évaluateurs émergents
Jérôme Gandin, Marie Gervais
LANCEMENT/LAUNCH
60
8h45-9h45
Influences et pouvoirs de l'évaluation à bâtir un monde meilleur
Dow Ornanong Maneerattana, Caroline Heider, Ghislain Arbour, Moderated By Benoit Gauthier
PLÉNIÈRE/PLENARY
9h45-10h
Lancement de l’appel à propositions pour l’édition spéciale en français de la Revue africaine
d’évaluation/African Evaluation Journal
Olivier Sossa, Marie Gervais
LANCEMENT/LAUNCH
61
27 MAI
MERCREDI WEDNESDAY
MAY
27
10h30-11h15
Contribution de l'AfrEA au développement des capacités de l'évaluation en Afrique
Serge Eric Yakeu Djiam
EXPOSÉ/PAPER
Même si diverses actions ont été menées au cours de ces dernières décennies pour le développement des capacités
des acteurs au processus d'institutionnalisation de l'évaluation, force est de constater qu'il existe encore de
nombreux défis et opportunités pour incorporer l'évaluation dans les cadres institutionnels en Afr ique. L'Association
Africaine d'Evaluation (AfrEA) coordonne environ plus de 30 associations nationales. L'un des principaux mandats de
l'AfrEA est sa contribution au développement des capacités en évaluation des membres pour améliorer la
performance des politiques publiques et privées. Cette présentation analyse les réalisations de l'AfrEA et passe en
revue les défis, les opportunités et les perspectives d'avenir dans le processus d'institutionnalisation de l'évaluation
en Afrique. Les décideurs et les praticiens ont reconnu le rôle central de l'évaluation dans les politiques de
développement.
Evaluation in sub-Saharan Africa: Adventures, Challenges, Reciprocity
Eugene Krupa, Kate Woodman, Clare Stead, Pascalina Chanda-Kapata, Anitha Menon
EXPOSÉ/PAPER
Our evaluations in Zambia have focused on capacity building and utilization, but also reciprocity in learning.
Initiatives included mLearning (primary schooling, training health workers), using digital devices to improve
community health services, reducing maternal death in childbirth, and reducing abuse and neglect of orphans. In
low-resource and cross-cultural settings, you want to do things well, and do so the first time. This means acting
respectfully, appropriately, efficiently, effectively, and with k nowledge translation, sustainability and scalability in
mind. It is challenging but, with experience and reflection, possible. It is essential to make strong partnership and
pursue high-level engagement with all relevant stakeholders, from policy makers to community residents and to
check for common understanding regularly. We describe the projects, key challenges, and solutions we found. We
ask that you bring your suggestions, tools, and recommendations to the discussion.
L’évaluation des politiques publiques en Afrique : le défi des aspects contextuels et culturels
Sylvain Nkwenkeu, Inoussa Kaboré, Marie Gervais
EXPOSÉ/PAPER
Les outils et approches méthodologiques qui sous-tendent la fonction d’évaluation sont le plus souvent construits et
affinés dans des contextes occidentaux et disséminés à travers le monde. Leur transférabilité et applicabilité aux
situations africaines, beaucoup plus complexes, demandent un effort de contextualisation qui prenne en compte les
aspects contextuels, socioculturels et économiques. Le panel discute de la prise en compte de ces dimensions en
distinguant l’approche prescriptive de type top-down qui repose sur un schéma linéaire et séquentiel d’une
approche plus systémique qui fait ressortir la circularité du processus de négociation entre acteurs multiples. Au
final, il convient de la nécessité d’alimenter une réflexion développementale afin de procurer du substrat cognitif et
normatif (représentations, valeurs/injonctions normatives, algorithmes) à l’évaluation, contri buant ainsi à consolider
le pont, tant décrié, entre la théorie et l’empirie.
The Evaluation of the 2009 Policy on Evaluation: Key Findings, Conclusions and Recommendations for
Developing a Renewed Policy on Evaluation
Anne Routhier, Suzannae Marshall
PANEL/PANEL
SCT Canada
62
11h15-12h
Plénière de clôture: Evaluation for a better world.... What does it mean for you? / Présentation des résultats
sur l’Agenda global 2016-2010
Maria Bustelo, Ernest House, Jonny Morrell, Rob Schwartz, Moderated By Astrid Brousselle
PLÉNIÈRE/PLENARY
12h-12h05
Passage du flambeau de l’Année internationale de l’évaluation
Benoit Gauthier
PLÉNIÈRE/PLENARY
7h-8h45
Les défis liés à la collecte et l’analyse de données et les solutions pratiques de l’audit interne et de
l’évaluation de programmes
Younes Mihoubi, Martin Saint-Louis, Pascal Théôret, Modéré Par François Dumaine, Associé, Pra Inc.
PETIT DÉJEUNER/BREAKFAST
8h45-10h
Défis et réalisations de l’évaluation pour une meilleure gouverne
Jim McDavid, Robert Lahey, Pierre Cliche, Moderated By Richard Marceau
PLÉNIÈRE/PLENARY
63

c2015 Projet de programme / Draft program

Transcription

Documents pareils

OFFICIAL PROGRAM PROGRAMME OFFICIEL

fashion show/défilé de mode

L`enseignement parental positif Positive Parenting ** OPEN TO ALL

Job Posting Adult Day Program Worker Community Support

Will code PHP for food

L`extrait vidéo que vous allez voir est tiré du documentaire Femmes

FOURNITURES POUR LA CLASSE DE C3 (2016/2017) Commande

Supprimer une partition CDFS (U3) - Homeres