Research within Evaluation: the case of Northern Ireland
1In K.Bush and C. Duggan (eds) Evaluation in the Extreme: Research Impact and Politics in Violently Divided Societies
India: Sage, 2015: 165-188
Contact details:
Colin Knox
School of Criminology, Politics and Social Policy University of Ulster
Shore Road Jordanstown Northern Ireland BT 37 OQB
Email: [email protected] Telephone: 028 90366667 Mobile: 07928208380
1 The author wishes to thank Dr. Kenneth Bush, Colleen Duggan, Dr. Kevin Kelly, John A. Healy and John R.
Healy for very helpful suggestions on this chapter.
2 | P a g e Word count = 8,295
3 | P a g e
Introduction
The starting point for this chapter is that it is written from the perspective of an evaluator of programmes in violently divided societies. As a result, the evaluation of research tends to be within the context of assessing the overall impact of a programme intervention in these societies. More specifically this chapter looks at how an evaluator assesses the contribution which research, as a distinct cluster of activities within a programme and running in parallel with other programme activities, contributes to the overall impact. In other words, a body of primary research is an integral part of the programme being evaluated, sits alongside the implementation of other multiple types of activities, and is expected to contribute to its impact. This is somewhat different than evaluating research outwith an intervention in at least two ways. First, research inputs to an evaluation are but one type of input to the process. In some cases, depending on the intervention, a parallel research track within a programme may be a significant component. In other cases, research can be a relatively small component and sit alongside a wide range of activities which comprise the intervention and directly deliver the objectives of the programme. Second, research within an evaluation can be done for different purposes. It may be about justifying the underpinning rationale for the intervention; it could be part of a formative assessment of the process/outputs; or it could be done in an effort to assess interim programme outcomes. In that sense the empirical point of reference for this chapter is exploring the interaction between research and evaluation. We do this in relation to a particularly contentious education programme intervention within the politically volatile environment of ‘post’ conflict Northern Ireland. Because the emphasis is on the evaluation of research from the perspective of an evaluator, the details of the case study are less important than how the evaluation of a cluster of research activities, among others, featured therein.
Hence, we provide a brief overview of the case study before moving to consider the substantive issues around the role and assessment of research as part of the wider evaluation process.
This chapter looks at the evaluation of research within programme evaluation under four key headings:
4 | P a g e
The evaluation of research on the theory of change which provides the rationale for the programme.
An evaluation of the research conducted by the delivery agents to assess the formative impact of the programme aimed at checking programme delivery.
The role played by the evaluator in assessing research within the context of programme evaluation.
The influence of research as one component in a programme of activities, and the contribution it makes to the overall programme objectives.
Before looking at each of these research components of programme evaluation, we outline brief details of the case study used in this chapter to investigate the topic in question.
The Case Study
The Northern Ireland education system is highly segregated along religious lines2 with almost 95% of children attending denominational schools: Maintained (Catholic) or Controlled (largely Protestant) state schools3. There are also a small-ish number of
2 Definitions of Catholic and Protestant (McGarry and O’Leary, 1995: 508-9): Catholic is a short-hand
expression for a believer in the doctrines of the Holy Roman Catholic and Apostolic Church; it is a synonym for an Irish nationalist. Protestant is a short-hand expression for somebody who is a believer in the doctrines of one of the may Protestant (including Presbyterian) churches in Northern Ireland; it is not a synonym for a unionist, although most Protestants are unionists; cultural Protestants are those who have Protestant religious backgrounds.
3The Northern Ireland Education system is highly segregated along religious lines denoted by various school management types as follows:
Controlled schools (mainly attended by Protestants) are managed by the Education and Library Boards through the Boards of Governors which comprise representatives of transferors (Protestant churches), parents, teachers and the education and library boards.
Voluntary (maintained) schools are managed by the Boards of Governors which comprise representatives of trustees (Catholic churches), parents, teachers and the Education and Library Boards. Responsibility for Catholic maintained schools rests with the statutory body, the Council for Catholic Maintained Schools (CCMS).
Voluntary (non-maintained) schools are mainly voluntary grammar schools managed by the Boards of Governors and represented by a cross-community umbrella organisation the Governing Bodies Association (GBA).
Integrated schools are schools which include pupils from both the Protestant and Catholic communities. The Department of Education accepts a balance of 70:30 (with 30% coming from whichever is the smaller religious group in the area) as the minimum required for a new school to be recognised as integrated.
There are also a number of Irish Medium schools (mostly in the primary sector) where children are taught through the medium of the Irish language. These are managed by the Boards of Governors and supported by Comhairle na Gaelscolaiochta (CnaG).
5 | P a g e integrated schools (n = 62) and Irish medium/language schools (n = 24). In the academic year 2010/11 there were 154,950 primary school children and 147,902 post-primary pupils in Northern Ireland, an overall school population of 302,852 pupils within some 1,200 schools (Department of Education, 2011). The first planned integrated school (Lagan College) was set up by parents in 1981 and the Department of Education was given a statutory duty to ‘encourage and facilitate’ the development of integrated education under the Education Reform (NI) Order 1989. The integrated education movement, according to its proponents, has experienced slow growth because numbers are capped within integrated schools, and requests to transform existing schools to integrated status are often refused by the Minister. The Department of Education claims it is difficult to facilitate the growth of a small integrated sector which can adversely impact on existing schools in the midst of a declining overall market where the school population is falling. In other words, the growth of integrated schools can simply displace children from other sectors and increase capital spending on the school estate for fewer pupils, in line with demographic trends.
In response to the slow pace of growth in integrated education yet an expressed demand from parents for greater mixing amongst school children from different community backgrounds4, two external funders, Atlantic Philanthropies5and the International Fund for Ireland6 set up the Sharing Education Programme (SEP) in 2007. Queen’s University Belfast (QUB), School of Education, began to work with 12 partnerships based on specialist schools (majoring in ICT, languages, arts etc) which collaborated on a cross-community basis to share classes and activities in order to improve education outcomes for pupils (Gallagher et al, 2010). The programme has an education curriculum focus, but because it is offered on a cross-community basis there are intended reconciliation benefits for participants, teachers,
4 In the Northern Ireland Life and Times Survey 2010 some 86% of respondents said they would like to see ‘a bit more’ or ‘much more’ mixing in primary schools; and 85% said they would like to see ‘a bit more’ or ‘much more’ mixing in post primary schools. The survey is based on a systematic random sample involving 1205 face- to-face interviews with adults 18 years or over.
5 Atlantic Philanthropies is a philanthropic organization funded by American Charles Feeney which aims to bring about lasting changes in the lives of disadvantaged and vulnerable people. It works on four main issues:
ageing, disadvantaged children and youth, population health, and reconciliation and human rights within seven countries: Australia, Bermuda, Northern Ireland, the Republic of Ireland, South Africa, the United States and Viet Nam.
6 The International Fund for Ireland was established as an independent international organisation by the British and Irish Governments in 1986. With contributions from the United States of America, the European Union, Canada, Australia and New Zealand, the total resources committed to the Fund to date amount to £628m / €753m, funding over 5,800 projects across the island of Ireland.
6 | P a g e parents and, in the long term, the wider community. The programme aims to demonstrate that sustained and ‘normalised’ collaborative contact will allow substantive relationships between peers and school communities to evolve across the religious divide (Atlantic Philanthropies, 2006). This creates interdependencies between the schools and reconciliation effects should result from children engaging in shared curriculum activities, teachers from across school sectors working together, and parents participating in school show-casing events (see theory of change in figure 1 below).
Figure 1: Theory of Change - Sharing in Education
The Queen’s University Sharing Education Programme completed a 3-year phase one in June 2010. The programme has been rolled out for a further 3 years and extended throughout Northern Ireland under two additional providers: the Fermanagh Trust (FT) and the North Eastern Education and Library Board (NEELB or PIEE7). These projects started in September 2009. Overall, the three projects included in the Sharing Education Programme represent an investment by funders (International Fund for Ireland and Atlantic Philanthropies) of over
£10.5m.
Although separately managed with project-specific outputs and outcomes, there are common overarching goals associated with all three projects. The overall aims of all 3 projects are as follows:
i. to increase the number of schools involved in shared education;
7 PIEE is the Primary Integrating/Enriching Education Project.
7 | P a g e ii. to increase the number of pupils involved in shared education;
iii. to help create more positive attitudes between Protestant and Catholic communities; and
iv. to demonstrate and raise awareness of the benefits from shared education in terms of integration and sustainability.
The range of activities undertaken in schools under the SEP includes:
Year 14 students completing Advanced Level (or ‘A’ level) subjects in cross- community classes
Year 12 students completing General Certificate of Secondary Education (or GCSE) subjects in cross-community classes
Jointly provided and accredited vocational training courses
Combined citizenship and personal development and mutual understanding (PDMU) classes
Science mentoring classes - primary schools children from mixed backgrounds attending science classes in a post-primary school
Collaborative ICT projects through face-to-face contact and web-based learning
The funders are attempting to lever change in education policy by collating evidence across the three areas of impact outlined in the SEP model above (education, economic and social benefits). Hence, the evaluation of the programme has attempted to answer key questions such as:
Societal well-being: does Sharing Education lead to greater tolerance, mutual understanding and inter-relationship through significant, purposeful and regular engagement and interaction in learning? And, does it lead to a reduction in barriers between school communities, create greater awareness of the benefits cross-sectoral working and the potential opportunities that sharing and collaboration can create?
Educational benefits: how (if at all) has Sharing Education improved the quality of education (however measured) for those involved? What additional benefits accrue beyond pre-existing single identity collaboration?
Economic case: is Sharing Education more cost-effective, providing value-for-money, when set alongside existing provision of education?
The author has been involved as an evaluator across the three projects which comprise the Sharing Education Programme. The remainder of the chapter focuses on the evaluation of:
the theory of change for sharing education; the research conducted by the delivery agents to assess its formative impact and check ongoing delivery against programme objectives;
8 | P a g e the role of the evaluator in assessing research; and, the overall influence of the research cluster within a programme which had multiple types of activities.
Evaluating competing theories of change
As an evaluator, one of the challenges which arises in terms of existing research on the nature of the intervention is the role that (s)he should take in considering this material. In other words, is it the role of the evaluator to assess the substantive merits/demerits of the research that informs programme design as a key element of any evaluation? If one accepts that the evaluator should have a role to play, it poses questions about the extent to which interrogation of the underpinning research within an overall intervention or programme of activities should feature in any evaluation of the outputs and outcomes. In addition, if one considers using the log frame rubric, so often the stock-in-trade of the evaluator, in which research will feature as one input to the intervention or programme of activities, how does one ‘weight’ the contribution of research as an input relative to other inputs such as the practical experience, knowledge and skills of those delivering the activities. What role therefore has research played in informing the underpinning theory of change and, by extension, the evaluation? Put simply, is it the job of the evaluator to question or problematise programme theory and design? Setting aside these important questions for the moment, let us assume that the role of the evaluator is explicit in evaluating the research component of any evaluation. What are the key considerations?
Research in and on violently divided societies, like many other areas of research, will often have competing theories of change. For example, in the case study topic which is the subject of this chapter, the role of integrated or shared education as a mechanism for social change is highly contested. One body of empirical research supports the whole idea of integrated schools as a way of addressing community divisions, typical of which is research by Hayes et al (2007) who conducted a detailed quantitative study on whether children experiencing a religiously integrated education had a significant effect on their political outlooks (see also McGlynn, 2007; Hargie et al, 2008; Stringer et al, 2009). This research concluded that attendance at an integrated school, either one formally constituted as integrated or religious school incorporating a proportion of pupils from the opposite religion, ‘has positive long-term benefits in promoting a less sectarian stance on national identity and
9 | P a g e constitutional preferences’ (Hayes at al, 2007:478). There is however an alternative literature which dismisses the whole idea of integrated education as a response to violently divided societies. McGarry and O’Leary (1995), for example, cite segregation as one of five key fallacies which constitute liberal explanations of the conflict in Northern Ireland.
Drawing on Bruce’s work (1994), they disparagingly describe attempts to break down segregation in this way as a ‘mix and fix approach’ espoused by the integrated education lobby who challenge stereotypes of the other religious group by tackling misconceptions and ignorance. In short, if segregation is the problem then mixing is the answer. McGarry and O’Leary reject this assertion outright. They argue that integrated education is impractical because residential segregation demands bussing children into hostile territory and mixed schools may simply exacerbate divisions on what separates groups rather than what they have in common. Whilst McGarry and O’Leary (1995: 856) supported the idea that ‘sufficient provision must be made for all those who wished to be schooled, live or work with members of the other community’ they argued that ‘many northern nationalists want equality and autonomy rather than integration’.
The key question here is whether it is the role of the programme evaluator to arbitrate on this research polemic, given that research forms the foundation of programme theory or should (s)he accept the underpinning theory of change and simply conduct the evaluation on that basis? Theories of change are a North American import into the field of policy evaluation in the United Kingdom and have been adopted as a way of addressing the problem of attribution by clearly specifying the links between inputs, activities, outputs and outcomes (Connell, J. P. and A. C. Kubisch, 1998). Yet the experience of UK evaluators using the theories of change approach has been that the involvement of stakeholders in developing and evaluating a relevant theory of change for a proposed intervention has not been entirely successful or difficult to achieve in practice (Bauld et al, 2005). Evidence from evaluations in the UK uncovered ‘principal’ and ‘elite’ ownership of theories of change where theory moved closer to ideology (Sullivan and Stewart, 2006: 180). Gaining consensus amongst stakeholders on an appropriate theory of change in a violently divided society when the focus of the intervention goes to the heart of what divides that society is likely to be even more difficult, implying a role for the evaluator in interrogating the programme
10 | P a g e theory. On the other hand, Sullivan and Stewart (2006: 194) warn against evaluator ownership where the theory of change is limited to, and dominated by, the evaluator and there is ‘no reference to the local agents who are responsible for delivering the policies’.
The author took the position that it was his role as an evaluator to interrogate the theory of change underpinning this intervention and therefore examined secondary research evidence as a way of validating the programme design. The key sources of evidence used (see table 1) can be categorised as: a deliberative poll amongst parents of school children on their attitudes to cross-community sharing; a scoping study on the economic benefits of sharing; yearly public attitude surveys on whether there was a demand for more ‘cross- community mixing’ in schools; and faith-based reports on the value of separate schools.
From this evidence the evaluator concluded that the theory of change which underpinned the sharing education programme had a sufficient evidence base to warrant a practical intervention of the type described above.
Table 1: Theory of Change – research components
Source of research Nature of the study Research strategy involved Newcastle University, Stanford
University and Queen’s University (funded by Atlantic Philanthropies)
Deliberative poll to gauge the opinion of parents of school- aged children about school collaboration within their area
Quantitative study, deductive and positivist
Oxford Economics (funded by the Integrated Education Fund)
Scoping study to assess the potential monetary benefits which could result from greater sharing and collaboration between schools. Makes the case for a wider follow-up study
Desk-based research using secondary analysis of data
Northern Ireland Life and Times surveys (funded from a number of sources, including Office of the First Minister and deputy First Minister and Economic and Social Research Council)
NI wide surveys which track attitudes to, inter alia, reactions to ‘more cross- community mixing’ in primary and post-primary schools
Yearly probability surveys of around 1,200 adults : positivist
Inclusion and Diversity in Catholic Maintained Schools
Articulation of the Catholic’s sector commitment to inclusion in their schools
Desk based experiential research written by School Principals in the Catholic sector
11 | P a g e
Evaluating formative research
Research strategies: A separate but related issue to competing theories of change is whether the evaluator ‘takes a position’ on the type of research that: (a) informs the underpinning theory in an evaluation; and, (b) is used by the delivery agents to provide a formative assessment of programme delivery/impact (Bryman, 2008). Theories of change often imply a deductive approach to research which begin with a set of theoretical assumptions, deduce a hypothesis(es), gather data to prove or disprove the posited hypothesis, and revise the original theory accordingly. An alternative approach to research is that the relationship between theory and research is primarily inductive where theory is the outcome of research. In other words, the process of induction involves drawing generalisable inferences from observations. The fluid and changing nature of context in violently divided societies might suggest that a more inductive approach is needed for formulating or testing theories of change. For example, the Organisation for Economic Co- operation and Development (OECD, 2008: 14) argues that many of the peace building interventions contemplated in violently divided societies tend to have relatively limited theoretical foundations, ‘including lack of agreed or proved strategies of how to effectively work towards peace’. Programmes may be based on no more than hunch or intuition of programme designers and/or donors on ‘what works’. Donors sometime ‘hide’ or ‘veil’ their theories of change for geo-political reasons or because the host government is hostile to donors’ theories of change and, in other cases, we simply don’t know what works. There are also epistemological considerations at play here. At a general level, there is the question of whether the social world can and should be studied according to the same principles, procedures, and ethos as the natural sciences (positivism), or should one respect the differences between people and the objects of the natural sciences which require the social scientist to grasp the subjective meaning of social action (interpretivism). For research in violently divided societies, the context, role of combatants in the conflict, political ramifications of interventions, and the need to find a resolution, all increase the importance of adopting an appropriate research approach.
If the evaluator has a preference for the role of theory and a specific epistemological orientation, then (s)he is more likely to favour either a quantitative or qualitative approach
12 | P a g e to the whole process: design of the evaluation, data analysis and the sort of knowledge that is valued as evidence. Quantitative methods lend themselves to a deductive approach to the relationship between theory and research or theory testing, and incorporate the norms of the natural scientific model or positivism. Qualitative methods are more synonymous with an inductive approach of theory generation and a preference for the ways in which individuals interpret their social world. Is it the role of an evaluator, who may have an individual preference for a particular research tradition, to make a judgement on this as part of his/her evaluation of the research which underpins the logic model and formative research on programme delivery/impact in and on violently divided societies? In short, if his/her own research background is deductive, positivist and quantitative, will this assume greater significance or importance in evaluating an intervention in a violently divided society than another evaluator whose research background is inductive, interpretivist and qualitative?
Research design: An example from the evaluation case study illustrates the dilemma for the evaluator in making judgements on the quality of research aimed at offering a formative assessment on programme delivery/impact. One type of intervention in the Sharing Education Programme supported cross-sectoral school activities between State (Protestant) and Maintained (Catholic) schools based on sustained contact between pupils in the delivery of the education curriculum through shared classes. This approach was predicated on the ‘contact hypothesis’ which asserts the value of inter-group contact in reducing hostility and improving inter-group relations under specified conditions (Pettigrew and Tropp, 2000). To assess the effectiveness of this approach, a questionnaire was completed by pupils on their experiences of shared classes, alongside a sample of pupils matched by age, religion and gender from the same schools but who were not involved in these classes.
Attitudinal data were gathered on trust, anxiety, perceived comfort and positive action tendencies towards those from a different religion and the differences between the participating and non-participating students assessed (Hughes, Donnelly, Gallagher and Carlisle, 2010) – see figure 2: model A.
13 | P a g e
FIGURE 2: EVALUATING PROGRAMME DELIVERY AND FORMATIVE IMPACT Model A: Regular contact between pupils
Compare this with an alternative approach adopted by a cluster of schools involved in SEP which did not wholly subscribe to pupil-to-pupil contact as the most effective way to promote reconciliation (or the reduction of hostilities and improved inter-group relations).
On the hunch or intuition of programme designers, the focus of their intervention centred on school principals and teachers as education leaders. Trust, they argued, needed to be established between schools leaders through shared principles, policies and practices.
Thereafter staff had to be supported and trained to work effectively in a new environment where cross-community schools become interdependent in the delivery of the school curriculum which, in turn, would enable pupil contact and sharing to take place. Integral to this approach was securing the endorsement of school governors and parents with the long- term aim of sustaining relationships beyond the life of the intervention. Evaluating this intervention involved in-depth interviews with school principals, teachers and governors and observations of pupils involved in shared classes. The qualitative data led to the generation of a testable theory: that collaboration through interdependency at the school leadership level is more likely to create the conditions for long term sustainability of pupil- to-pupil contact (Knox, 2010a) – see figure 2: model B.
Baseline data
Random allocation of
pupils
Intervention group
Control group
Sharing Education Programme
Attitudinal data on trust,
anxiety, comfort with
other community
Attitudinal data on trust,
anxiety, comfort with
other community
14 | P a g e
FIGURE 2: EVALUATING PROGRAMME DELIVERY AND FORMATIVE IMPACT Model B: Training School Leaders
In summary, one implementation approach was to see the pupils as the focus of activities (figure 2: model A) and the other was to work with school principals and teachers (figure 2:
model B). Evidence gathered on the success of the former was deductive, positivist and quantitative using a quasi-experimental design. Evidence on the latter was inductive, interpretivist and qualitative, largely based on data gathered through semi-structured interviews and observations. In these examples, what is the role of the evaluator? Having interrogated the theory of change which underpinned the programme, there are alternative implementation processes. Here the evaluator was faced with quite different ways of attempting to deliver programme objectives and his role was to assess the quality of research conducted by the projects in their formative assessment of the programme. The evaluator needs to be wary of making judgements on the nature/quality of this formative research based on his/her preferred research design (either inductive or deductive). In this case, looking at the quality and usefulness of the evidence gathered, the evaluator concluded that the quasi-experimental approach (figure 2: model A above) did not capture the complexity of attitudinal change in participating and non-participating pupils with the
15 | P a g e sharing education programme. Rather, the richness of qualitative data gathered through principals, teachers, parents and governors (figure 2: model B above) offered much more useful insights into the ongoing delivery and formative impact of the programme. Table 2 summarises the research sources which the evaluator used in making judgements on programme delivery and formative impact.
Table 2: Formative evaluation – research components
Source of research Nature of the study Research strategy involved Queen’s University Belfast,
School of Education (part- funded by the Economic and Social Research Council and Atlantic Philanthropies)
On-line survey of head teachers, face-to-face
questionnaires with pupils and in-depth case studies of schools involved in collaborative activities
Mixed methods approach – interviews with teachers, classroom observations, and survey work with teachers and participating pupils:
interpretivist and positivist Schools participating in Sharing
Education Programme
Views of school principals and teachers involved in delivering shared education programme alongside opinions of school governors and parents
Qualitative, inductive and interpretive
Popular press Editorials, opinion pieces and letters to a range of local and regional newspapers
Informed popular opinion and public reactions to SEP Hansard/Official Report Debates, motions or
parliamentary questions in Northern Ireland Assembly on shared/integrated education
Secondary research drawing on existing studies supplemented by party-political opinions.
Witnesses called to give
evidence to statutory education committee on the
implementation of SEP
Robustness of the research: Directly linked to the above discussion on different research designs, the role of the evaluator must also include some evaluation of the robustness of research conducted by those delivering the programme to assess its formative impact.
Research in and on violently divided societies poses particular problems here around reliability, replication and validity. For example in the case study under consideration here, the measures used to assess whether shared education changed the attitudes and behaviour of pupils to the ‘other’ community may be unstable over time. There can be problems of internal reliability in attitudinal scale items used in the questionnaire administered within intervention and control schools. One important component of the
16 | P a g e research activities associated with the implementation of the intervention involved observations within schools conducted by a number of researchers. This can give rise to inter-observer consistency as to how these observations can be categorised and analysed.
Because of the context of violently divided societies it will be difficult or impossible to replicate the findings across space and time, not least in a school setting where data are gathered from a cohort of pupils who move through the school system. Linked to the concept of reliability is the issue of measurement validity. How can we be sure that the measurements involved in testing tolerance towards the ‘other’ community in the case study here capture that complex concept? Moreover, can we be sure that the intervention of shared education was causally linked to greater tolerance amongst young people who participated in these programmes – the whole issue of internal validity. The evaluator of the research activities associated with programme delivery must therefore exercise an explicit role in judging the quality of research which (s)he is evaluating. This is more important in violently divided societies because of the contested nature of interventions normally associated with ameliorating the source of the division.
Role of evaluator in judging research
If one accepts that researchers bring personal values and bias to the process of social research, then it is incumbent on the evaluator to consider the source/nature of research and the motivation of the researchers who produce the knowledge that eventually finds its way into the programme design. In violently divided societies this is particularly important because those issues which are the basis of division (ethnicity, religion, language, national identity etc) will attract the attention of different researchers with values which reflect their own biases and are likely to straddle the division(s) in question. In the case example in this chapter, for example, the Integrated Education Fund (IEF) commissioned research entitled Developing the Case for Shared Education whose objective was ‘to assess the availability of information required to properly understand the fiscal implications (costs and savings) of alternative budget scenarios and a move towards a more shared education delivery system’
(Oxford Economics, 2010: 2). To this evaluator it was clear from the commissioning source (the Integrated Education Fund), the title and the objectives of the study, that emergent research would favour shared education. Equally, the funders (the International Fund for
17 | P a g e Ireland (IFI) and Atlantic Philanthropies) of the Sharing Education Programme, the subject of this case study, have organisational values which favour a more integrated society in Northern Ireland. The mission statement of IFI (2010) is to ‘tackle the underlying causes of sectarianism and violence and to build reconciliation between people and within and between communities throughout the island of Ireland’. This is an explicit articulation of what the organisation aims to achieve. An evaluator is therefore clear about the values and biases of the funded interventions emanating from this source. The values outlined here can also have implications from the choice of the evaluation questions to pose, through the design of the evaluation, to analysis and interpretation of data.
Does all of this imply that the evaluator should be someone with substantive or specialist expertise in the policy field who has an intimate knowledge of the context and its stakeholders? If so, then the generalist evaluator becomes obsolete. Equally, should an evaluator have some affinity with the funders’ values? What, for example, are the implications if shared education actually divides communities and increases violence? These questions are related to a discussion on the independence of the evaluator – is a generalist evaluator more likely to be independent than a specialist evaluator when making judgement on the rationale for a programme, a question which has received attention in the evaluation literature. Michael Scriven (1996), for example, is unequivocal in his views that the evaluator must be completely independent when making judgements within an evaluation and guard against being incorporated as an advocate of the programme (s)he is evaluating.
Patton (2008: 500-01), on the other hand, adopts an alternative position which includes two different roles for the evaluator: (a) the evaluator-facilitator who ‘facilitates others’
interpretation, judgements and recommendations’; and, (b) the evaluator who renders his/her own interpretation ‘either separately of as part of the interactive process’. The evaluator can move back and forth between these roles in the active-reactive-interactive- adaptive model of utilization-focused evaluation which Patton has pioneered. Although articulating this polemic in the literature is interesting, it offers limited normative guidance on whether theory of change and assessing formative research findings are best evaluated by a specialist or generalist evaluator, except perhaps to imply that the former may be less likely to adopt an ‘evaluator-facilitator’ role given his/her knowledge of ‘what works’.
18 | P a g e In its simplest form, one approach to evaluation is to use a linear model of: inputs → processes → outputs → outcomes, with the latter being seen as the ultimate rationale for a programme of activities. In order to judge whether the outcomes of an intervention have been achieved, the evaluator simply refers back to its original goals which become the source for deriving programme outcome measures. If the evidence based on the outcome measures supports the attainment of programme goals, the intervention can be deemed successful (Dahler-Larsen, 2005). The generalist evaluator requires good research skills to gather and interrogate data consistent with this approach, rather than a specialist knowledge of the substantive topic of the evaluation. On the other hand, violently divided societies are a very different evaluation milieu where the ‘normal’ processes of data gathering, analysis and interpretation don’t apply. Interventions in these societies can often be funded by external and well-intentioned benefactors (as is the case in the case study in this chapter) who may support popular activities that demand a specialist evaluator capable of not only dealing with the violent context in which they are delivered but also bringing his/her research expertise to bear on judging the quality of the intervention.
The influence of research
As mentioned at the beginning of this discussion, the key role of any evaluator is to assess the impact of the intervention which (s)he is assessing. An important consideration must therefore be what contribution did the research make to the impact of an initiative and is that any different for evaluations conducted in and on violently divided societies? There is evidence that practitioners in these societies are sceptical about the merits of evaluation and hence how research is treated will be a key consideration vis-à-vis other inputs to the intervention process (OECD, 2008). But the common problems of evaluation, namely the attribution dilemma (that the achievement of an outcome can be directly and solely attributed to a single intervention) and the counterfactual, could be seen as more crucial in evaluating interventions in violently divided societies – not least because there is limited evidence of what works. There are other dilemmas for the evaluator of research. The evaluation may uncover robust and compelling research but find that it has been poorly used within the intervention. In the case study example in this chapter, advocacy skills in promoting social change in Northern Ireland are still under-developed. This is because, until
19 | P a g e recently, locally elected representatives had limited public policy leverage. During direct rule for Westminster (1972-99, save for a short interlude), British Ministers and unelected civil servants made the major policy decisions (Knox, 2010b). Although powers were devolved to the Northern Ireland Assembly in 1999 it was highly unstable until 2007 when a mandatory power sharing coalition was established. Hence, although a robust body of research may exist, convincing the policy makers and delivering social change requires a different set of skills.
This also raises a question about the extent to which the impact of an intervention is necessarily evidence-informed and that research is therefore crucially important. The subject of this chapter is about an intervention in a highly segmented system of education aimed at promoting greater sharing and collaboration between schools from different community backgrounds. The ultimate test of its impact is whether there are better education outcomes for pupils and, more widely, if strong positive reconciliation effects result. Will the Department of Education therefore incentivise sharing over separation in schools? This requires politicians to endorse a fundamental change in the way in which schools are structured, funded and operate on a daily basis. The First Minister, in a speech in October 2010, created a huge political momentum when he described the education system in Northern Ireland as a ‘benign form of apartheid which is fundamentally damaging to our society’ and argued for a carefully planned and ‘staged process of integration’ (Robinson, 2010: 1). This came on the back of comments by the Secretary of State for Northern Ireland telling Conservative Party members that the British taxpayer should not have to foot the bill for a system of parallel schools. He argued that separate schooling ‘is a criminal waste of public money. We cannot go on bearing the cost of segregation and I don’t see why the British taxpayer should go on subsidising segregation’ (Paterson, 2010: 4).
These two important statements by the Secretary of State and the First Minister opened the door for a political debate on the topic. On 22nd November 2010 the Northern Ireland Assembly debated (under Private Members’ business) the topic of integrated and shared education and as a result of the debate passed the following motion:
This Assembly… believes that the current system of education is unsustainable, recognises the economic, educational and social benefits that can come from
20 | P a g e integrated and shared education; and calls on the Minister of Education to actively promote a system of integrated and shared education throughout Northern Ireland (Hansard, Official Report, 2010)
During the debate in the Assembly reference was made by several MLAs to the Sharing Education Programme as a successful model of sharing which should be considered by the Minister and her Department of Education.
Context is important when considering the influence of research. The political context of Northern Ireland and other violently divided societies is such that all policy interventions could be viewed as zero-sum. For example, an increase in integrated or shared education can be interpreted as an attack on the Catholic faith-school tradition. The evaluator needs to be acutely aware when conducting evaluations that his/her findings will be viewed within this win/lose framework and one can become personally associated as an exponent or critic of the central intention of any intervention. As an evaluator from a Catholic community background this evaluation presented some moral struggles – could one be sufficiently dispassionate when evaluating a programme which espouses a more integrated or shared schooling sector when personally educated in a system which promoted the Catholic school ethos? On a wider political level, this type of education programme challenged the huge influence which the churches (Catholic and Protestant) historically exert on the school system. All of which suggests that programme evaluation is context-bound. In fact, context mattered enormously in this case study because of the changing political and education policy environment. Key political antagonists (DUP and Sinn Féin) had reached political accommodation to share power in a devolved government and were looking for policy areas on which they could achieve consensus. A reduction in public sector spending had become a key economic priority for the UK Government, one element of which was a significant cut to block grant assistance to Northern Ireland. The school population was in decline, there was an over-supply of school estate, and the system of parallel education provision (Controlled and Maintained schools) could no longer be sustained. In addition, the Catholic Church had suffered bitter criticism over the role its senior clerics had played in concealing child abuse by priests and, as a result, the whole idea of a distinctive Catholic school ethos can under scrutiny. This confluence of events offered a more receptive political and public policy
21 | P a g e context for, at the very least, new ideas of how education could be delivered. In short, single identity education provision was ripe for reform.
The key question for the evaluator is whether the intervention in shared education had simply caught the wider political tide or was it instrumental in creating it? Unsurprisingly politicians in the Assembly debate (above) were not particularly interested in the detail of the research evidence but rather that it was broadly supportive of shared education. It seems reasonable to suggest that the wider political environment which led to a consociational model of power sharing created a context in which ‘bold’ policy interventions could flourish. Was the evidence from the evaluation of the shared education intervention simply waiting for political endorsement or was the emerging political commitment awaiting empirical support? It is difficult to assess the direction of association or indeed whether such a simple relationship exists exclusively between these two factors or variables. Schools collaboration may have more to do with the retrenchment in public expenditure on education and the excess of school places (empty desks) rather than any cross-community or reconciliation motives. An evaluator has the difficult task of assessing the influence that research might have in the context of significant political momentum for change in the way in which education is delivered. The evaluator also accepted that research is but one component within multiple activities which comprise the Sharing Education Programme and social change is influenced in many different ways through advocacy, media campaigns, opportunism, political support, random events etc. Importantly, research can often be far down the results chain and its direct association with the aspiration of a reconciled community in Northern Ireland seems tenuous. We summarise the key elements of the narrative in table 3 below.
22 | P a g e
Table 3: Evaluating research in programme evaluation
Evaluating theory of change
Evaluating formative research on programme delivery/impact
Role of evaluator in judging research
Influence of research on programme impact
Role of evaluator in questioning theory of change:
Shared schooling
‘Mix and fix’
approach
Separate but equal communities
Validate programme design from secondary research evidence
Evaluator makes an assessment of formative research evidence based on the following:
Choice of research strategies: deductive, inductive
Choice of research design: quasi-
experimental, qualitative
Robustness of the research – reliability, replication and validity
Transparency around values and biases of programme
commissioning body
Generalist or specialist evaluator?
Independence of evaluator – openness in personal values and biases
‘Weight’ of research relative to other programme inputs
Importance of political context and
endorsement
Relationship between research and policy change. Direction of association – is policy change evidence informed?
Conclusions
What are the challenges which face an evaluator when tasked with evaluating research in a violently divided society? It is precisely because evaluations in divided societies will often be about contested interventions that there is a need for reflexivity on the part of the evaluator. This will demand a number of things. The evaluator should, as a matter of course, articulate his/her own values and biases. The author of this chapter is from a Catholic community background, attended a single sex Catholic voluntary grammar school in Northern Ireland, sent his children to a mixed gender State (Protestant) grammar school, and has conducted research which espouses a more integrated society in Northern Ireland.
This should be made clear in any evaluation. Going beyond self-reflection and because of the contested nature of interventions in violently divided society, the evaluator should indicate explicitly his/her research values. This will involve subscribing to a particular research tradition. The author of this chapter has a predisposition for quantitative methods, deductive research and favours positivism. Such reflexivity, while stripping bare the essential values and biases of the evaluator will also make clear his/her starting point in an evaluation. The evaluator also has a key role to play in interrogating the theory of change.
23 | P a g e Even where an intervention is based on the hunch or intuition of programme designers working in the field, the role of the evaluator should be to take this as the starting point, interrogate it, rather than being critical of the absence of a clear underpinning theory for the intervention and look for evidence of its success or failure. To do otherwise would be to bring a normative stance to policy evaluation which simply reinforces the values and biases of the evaluator. In the circumstances of already contested interventions in violently divided societies, this simply limits the scope for evaluation and supports the notion (described by the OECD) that we are unclear about ‘what works’ in peace building.
If, as we argue, it is the role of the evaluator to interrogate the theory of change which underpins an intervention, (s) he also has a role in judging the robustness of the research which seeks to operationalise it. Notwithstanding the difficulties within the context and field of enquiry (violently divided societies) the evaluator should not accept lower standards of research. Perennial research issues of measurement, replication, causality and the counterfactual, for example, are challenging whatever the field of evaluation. The fact that these are more demanding in violently divided societies should not be a reason to lower standards of research but rather an opportunity to be imaginative and creative about ways in which measurements issues can be improved. In fact, the role of the evaluator should be to press for higher standards of research precisely because evaluation stakeholders are often highly sceptical about evaluation and equally critical of judging ‘what works’ in these societies. The evaluator can play an important role in judging the quality of research conducted by those delivering programmes and aimed at making a formative assessment of impact and checking against delivery targets. His/her role is to interrogate this research which may influence programme delivery in the first instance but ultimately contributes to better programme impacts.
A key challenge for the evaluator is to assess the ‘weight’ ascribed to research as one amongst several inputs in any intervention process. If research is a key component then it drives the process of evaluation, and may demand someone with specialist expertise in the intervention. Such a high degree of specialisation might be difficult to justify, push up the costs of evaluation and create supply side problems in accessing specialist evaluators. What is clear, however, in evaluating research in violently divided societies, because the
24 | P a g e interventions are often about those issues which are the source of the division, is the need for greater transparency in the evaluation process. This should include: listing the nature, source and funding for the research; articulating personal values and biases on the part of researchers and evaluators; and a clear articulation of the contested political context in which the intervention takes place. In short, there should be greater interaction between research and evaluation.
25 | P a g e
References
Atlantic Philanthropies (2006) Promoting an Inclusive Education System: Belfast: PA Consulting Group.
Bauld, L., K. Judge, M. Barnes, M. Benzeval, M. Mackenzie and H. Sullivan (2005) ‘Promoting social change: The experience of Health Action Zones in England’, Journal of Social Policy 34(3): 427–45.
Bruce, S. (1994) The Edge of the Union: the Ulster Loyalist Political Vision. Oxford: Oxford University Press.
Bryman, S. (2008) Social Research Methods (3rd ed.). Oxford: Oxford University Press.
Connell, J. P. and A. C. Kubisch (1998) ‘Applying a Theory of Change Approach to the Evaluation of Comprehensive Community Initiatives: Progress, Prospects and Problems’, in K. Fulbright-Anderson, A. C. Kubisch and J. P. Connell (eds) New Approaches to Evaluating Community Initiatives, vol. 2, Theory, Measurement and Analysis, pp. 15–44.Washington, DC: Aspen Institute.
Dahler-Larsen, P. (2005) ‘Evaluation and Public Management’ pp. 615-642 in E. Ferlie. L.
Lynn and C. Pollitt (eds.) The Oxford Handbook of Public Management. Oxford: Oxford University Press.
Pollitt (eds.) The Oxford Handbook of Public Management. Oxford: Oxford University Press.
Department of Education Northern Ireland (2011) Enrolments at Schools in Northern Ireland.
Bangor: Department of Education.
Gallagher, T., Stewart, A., Walker, R., Baker, M. and Lockhart, J. (2010) Sharing Education through Schools Working Together. Shared Space, Issue 10. Belfast: Community Relations Council.
Hansard, Official Report (2010) Private Members’ Business: Integrated and Shared Education. 22nd November.
Hargie, O., Dickson, D., Mallet, J. and Stringer, M. (2008) ‘Communicating Social Identity: a study of Catholics and Protestants in Northern Ireland’ Communication Research 25 (6): 792 – 821.
Hayes, B., McAllister, I. and Dowds, L. (2007) ‘Integrated Education, Intergroup Relations and Political Identities in Northern Ireland’ Social Problems 54 (4): 454 – 482.
Hughes, J., Donnelly, C., Gallagher, T. and Carlisle, K. (2010) School Partnerships and Reconciliation: an evaluation of school collaboration in Northern Ireland. Belfast: Queen’s University, School of Education.
26 | P a g e International Fund for Ireland (2010) Mission statement, accessible at
http://www.internationalfundforireland.com/strategy
Knox, C. (2010a) Sharing Education Programme: Views from the White Board. Jordanstown:
University of Ulster.
Knox, C. (2010b) Devolution and the Governance of Northern Ireland. Manchester:
Manchester University Press.
McGarry, J. and O’Leary, B. (1995) ‘Five fallacies: Northern Ireland and the liabilities of liberalism’ Ethnic and Racial Studies, 18 (4): 837 – 861.
McGarry, J. and O’Leary, B. (1995) Explaining Northern Ireland: Broken Images. Oxford:
Blackwell Publishers.
McGlynn, C. (2007) ‘Rhetoric and reality: are integrated schools in Northern Ireland really making a difference?’ Irish Educational Studies: 26 (3): 271 – 287.
Northern Ireland Life and Times Survey (2010) results accessible at:
http://www.ark.ac.uk/nilt/2010/
Organisation for Economic Development and Co-operation (2008) Guidance on Evaluating Conflict Prevention and Peace Building Activities. Paris: OECD.
Oxford Economics (2010) Developing the Case for Shared Education, Scoping Paper. Belfast:
Oxford Economics.
Paterson, O. (2010) ‘Schools strategy is a criminal waste of public money’. Belfast Telegraph 6th October: page 4.
Patton, M. Q. (2008) Utilization-Focused Evaluation (4th edition). London: Sage.
Pettigrew, T. and Tropp, L. (2000) ‘Does intergroup contact reduce prejudice: recent meta- analytic findings’: 93 – 114, in S. Oscamp (ed.) Reducing Prejudice and Discrimination.
Mahwah NJ. Lawrence Erlbaum.
Robinson, P. (2010) ‘First Minister argues for a single education system in Northern Ireland’.
Peter Robinson’s speech to Castlereagh Council, 15th October – accessible at:
www.dup.org.uk
Scriven, M. (1996) ‘Types of Evaluation and Types of Evaluators’, Evaluation Practice, 17:
151-162.
Stringer. M., Irwing, P., Giles, M., McClenahan, C., Wilson, R. and Hunter, J. A. (2009)
‘Intergroup contact, friendship quality and political attitudes in integrated and segregated
27 | P a g e schools in Northern Ireland’ British Journal of Educational Psychological Society, 79: 239- 257.
Sullivan, H. and Stewart, M. (2006) ‘Who Owns the Theory of Change? Evaluation, 12(2):
179-199.
