Journal of Research and Reflections in Education June 2007, Vol. 1, No. 1, pp 28 – 56
http://www.ue.edu.pk/jrre
Challenges of Improving Teachers’ Assessment Practices
Duishon Alievich Shamatov Abstract: This article reports the outcomes of a qualitative case study
which explored how school teachers develop their assessment while attending an in-service teacher education programme. Semi-structured interviews and document analysis were two methods used to explore the Master of Education students at Aga Khan University Institute for Educational Development (AKU-IED). These students come from developing countries in Central Asia, South Asia and East Africa. Prior to joining AKU-IED, most of these students have had experienced assessment practices characterized in my study as ‘traditional’, teaching and learning have been geared towards examinations and tests. The selective purpose of assessment selects and rejects people, and social reproduction is maintained by the traditional assessment. Additionally, traditional assessment has often caused great deal of psychological discomfort and elements of ‘unproductive competition’ reflecting on extrinsic reward in schooling. At AKU-IED, the M.Ed. students have a very intensive transformative learning experience. They are encouraged to critically examine their existing educational philosophy, including assessment notions. As a result, many students re-modify their assessment theories and practices. After the completion of the M.Ed. program the students are potentially in a position where they are able to influence assessment practices, to varying degree, when they return to their home work environments.
Keywords: qualitative approach, paradigm shift, assessment practices, reconceptualization, change
Introduction
Assessment is a process of collecting, synthesizing, and interpreting information on students’ performance to measure their progress (Airasian, 2005; Harris & McCann, 1994). Based on information obtained about the students’ progress, decisions are made about students, curricula, programmes and educational policies (Capper, 1996; Popham, 2001). Assessment is an integral part of the teaching and learning process, and it can be used to enhance the process rather than hinder. As Murphy and Torrance (1991) argue,
“Wherever learning takes place, or it is intended that it should take place, then it is reasonable for the learner, the teacher and other interested parties to be curious about what has happened both in terms of the learning process and in terms of any anticipated or unanticipated outcomes” (p.3). Traditional assessment approaches hinder educational reforms and have negative washback effects in teaching and learning. There is a need to re-examine existing notions of assessment of teachers and help them develop alternative approaches to assessment. Teacher education with particular emphasis on assessment practices can serve as a vehicle for improvements in education elsewhere.
I conducted a qualitative study at the Aga Khan University Institute for Educational Development (AKU-IED). AKU-IED was established in 1993 and its aims was to address “irrelevant curricula compounded by ineffective assessment which in turn fostered rote learning and passive student roles” (Phase 2 Proposal, AKU-IED, 1997) by improving the performance of teachers through professional development and school improvement. A major programme at AKU-IED is the two-year M.Ed. degree programme. M.Ed. students at AKU-IED come from a range of developing countries such as Afghanistan, Bangladesh, Kenya, Kyrgyzstan, Syria, Tanzania, Tajikistan, Uganda, as well as from different regions of Pakistan. The study was conducted with a group of Master of Education students. To study how M.Ed. students learn about assessment practices I explored their previous experiences with assessment practices; the extent and ways they have learned while at AKU-IED; the factors that help them learn;
and the implications of their learning for subsequent professional practice. I employed semi structured in-depth interviews with three cohorts of M.Ed. students and their faculty instructors with a total number of 28 people.
Theoretical Framework
Traditional assessment approaches based on paper-and-pencil examinations and tests dominate many educational institutions all over the world. As Capper (1996) states,
“many countries complain that their education system is so exam ridden that the entire teaching and learning process is geared to passing the exam and getting the good marks needed for entry to higher levels of education” (p.1). The purposes of assessment have also been seriously questioned in recent decades. In a strident critique, Holt (cited in Murphy & Torrance, 1991) observed:
There are two main reasons why we test children: the first is to threaten them into doing what we want done, and the second is to give us a basis for handling out the rewards and penalties on which the educational system - like all coercive systems must operate. The threat of a test makes students do this assignment. The outcome of the test enables us to reward those who seem to do it best. The economy of the school, like that of most societies operates on greed and fear. Tests arouse the fear and satisfy the greed. (p.8)
Other writers commented on the selection function of traditional assessment.
For example, Wilby (cited in Murphy & Torrance, 1991) stated that UK secondary education is organized to select a very small number of students to go to university and further. He observed, “For their sake, all our children are being put through an over- blown, over-academic syllabus, in which the dominant experience, for the majority, is one of failure, not of achievement” (p.13). Selection, it is frequently noted, is not necessarily an objective process but rather has implications for social reproduction and for introducing self-fulfilling prophesies on students’ futures and self-esteem. Harris and McCann (1994, p.2) suggest that, “The more able ones may even enjoy these experiences [tests], as they can assert their superiority over the rest of the class. However, many students feel anxious, worried, and inadequate. There is often great pressure on them to succeed and if they do not, they become branded as failures”. Murphy and Torrance(1991) observe that the selection purpose is “the mould of the narrow academic curriculum geared to the selection requirements of higher education and the needs of a society obsessed with selecting an elite minority through the process of schooling” (p.12).
The social reproduction and/or ‘meritocratic’ functions of assessment have a substantial history in the now ‘developed’ countries. Patricia Broadfoot (cited in Torrance 1995) comments:
When educational assessment first began to be widely used in the 19th century, this was in response to the institution of mass educational provision and the associated industrial economics of that era. The pressing need to find mechanisms of selection that would be both socially acceptable and identify the ‘best’ candidates led to a premium
being put on assessment techniques that appeared to be fair, objective, that had high levels of reliability.
Murphy and Torrance (1991) further state that “What is also necessary, however, is the need to ensure that these particular purposes [selection and accountability], or functions of educational assessment, are not allowed to dominate both the entire school curriculum and the complete range of assessment procedures that are associated with it” (p.14).
The negative outcomes of the selection function of traditional assessment are also apparent in developing countries. For example Singh (1993, p.236), writing with respect to India, makes an argument which is similar to that of Holt and of Murphy and Torrance. Singh draws a powerful connection between selection and social consequences, as well as resource allocation. He argues that,
Selection tests are probably what most people think of when they talk about assessment. Actually, it is often somewhat euphemistic to call them ‘selection’ tests. For the majority of candidates many such tests function rather as rejection tests. Selection (and rejection) is necessary because no country believes it can afford to give every citizen all he might desire in the way of education. ... In many developing countries, places are in short supply even for secondary education and most children will fail to be selected. One of the assumptions implicit in selection tests for education is that only the brightest, most promising, and patently talented should be funded to continue.
Selection functions of assessment, therefore, act to promote competitiveness among students. Competition may be appropriate and necessary. It certainly reflects a social reality. However, it becomes problematic when competition becomes an end in itself, or when it distorts the essence of education or when it serves the interests of certain social groups to the detriment of those who may already be disempowered or disadvantaged. ‘Rejection’, to use Singh’s term, is in this sense not so much an objective judgment but an aspect of processes which do not necessarily reflect equity in their application or their intent. “Unfortunately this competition creates more losers than winners” (Harris & McCann, 1994, p.2).
In addition to selection-related issues, many other problems and issues associated with educational systems in western or developed contexts have become particularly important in developing country contexts. There is evidence that some of the more negative practices have been adopted, and sometimes worsened, in educational systems in, for example, South Asia. This has been to the detriment of the quality of teaching and learning. Commenting recently on the Indian higher education system, Sinha (1993) observes that, “It is beyond doubt that the greatest evil from which the system of Indian university education suffers is that teaching is subordinated to examination and not examination to teaching”. In a summary which is typical of critiques of assessment practice and negative impact on education in South Asia, Sinha (p.247) comments:
... the shortcomings of the traditional examination system:
memorization was over-emphasized, coverage of the pupil growth process was limited, mercurial subjectivity was omni-present;
inadequate use was made of test results; and receptivity to reform and change was discouraged. The traditional situation not only nullified all the criteria of a sound evaluation system, but also encouraged the adoption of slipshod methods of teaching and learning. Besides, it provided wide openings for malpractices to enter.
In Pakistan, Farooq (1994, p.17) similarly comments that, “Too much importance is given to scores in academic examinations. As a result, there is enormous corruption in the examination system” (author’s emphasis). He claims that students, teachers, parents and heads of institutions are equally responsible for this situation.
Farooq’s reference to shared responsibility is an area which is of considerable relevance to the present study. The proposition that multiple stakeholders support a system which may act counter to the interests of educational quality and opportunity is an area which is investigated. Analyzing the current situation of the quality of examination in Pakistan, Bethel et al (1995, p.1) state that it is,
... extremely low. Many question papers contain errors in subject content, language, and technical construction. In addition, they focus on a narrow range of low-level skills and are dominated by the content
of the approved textbooks. In consequence, the examinations have a negative effect on the educational process in Pakistan.
The phrase is sometimes used that assessment is the ‘tail that wags the dog’.
That is, assessment has a disproportionate influence over the curriculum and the teaching and learning in schools, and perhaps higher education. As Fisher (1991) states “The assessment process itself should not determine what is to be taught and learned. It should be the servant and not the master of the curriculum, yet it should not simply be a ‘bolt-on’
addition at the end. Rather, it should be an integral part of the educational process, continuously providing both ‘feedback and feedforward”.
Other numerous writers have identified practices of teaching and learning in schools which are relatively ineffective but which often reflect the pressure on teachers and others to maximize preparation for an examination-oriented assessment system, a version of the so-called ‘washback effect’. These practices include: rote learning and memorization; lower order thinking; reliance on textbooks; transmission models of teaching; coverage of prescriptive syllabus; primary focus on measurable skills; a focus on making judgments. As Harris and McCann (1994, p.3) summarize:
Yet another reason for negative attitudes to assessment among students is that, rather than give them the opportunity to show what they have learnt, it tries to catch students out, to reveal what they have not learnt.
Learners also feel alienated by assessment because they have no role in it, apart from as passive participants. For many learners in this situation, assessment must seem arbitrary and at times even unfair.
Sometimes they get on with their teacher, sometimes they do not.
Sometimes they are lucky and revise the right material for a test, sometimes they are unlucky.
However, it does need to be stressed that the critique outlined above is not intended to imply that traditional assessment has no place or is invariably dysfunctional.
Clearly, this is not the case, not least because traditional aspects of assessment, including for example selection and accountability functions, are and need to be part of the educational process. Some of the common distinctions between approaches to assessment are introduced above, for example, formative (as contrasted with summative),
self-assessment (as against normative). Macintosh and Hale (1976) illustrate the purposes which may underpin decisions to adopt several of the assessment strategies above. They refer to purposes as diagnosis, evaluation, guidance, grading, selection and prediction.
As the focus of this study is the paradigm shift in assessment from traditional to alternative forms which is undertaken at AKU-IED. Much of this study is concerned with the process of this shift, whereas most literature and research focus on either one approach or the other, or characterize both. The following quotation begins to address important issue for the present study of the process of transformation from one assessment approach to another.
The transition from traditional instructional and grading practices to an authentic system of assessment represents a significant educational transformation. During this transition, classrooms are changing from a teacher-centered testing culture, where students work individually and learning is done for the test results, to a collaborative assessment culture, where assessment takes many forms, reaches multiple audiences, and distinctions between learning and assessment are blurred (Barron & Boschee, 1995, p.77).
The transition from one form of assessment to another is not easy. For example, Sinha (1993) states that, “Reform is invariably a difficult task to accomplish, because the traditional procedures and practices have become deeply rooted in the system.”
Methodology
A qualitative approach was used to study the master of education students’
experiences of learning assessment (McMillan & Schumacher, 1997; Sowell, 2001).
Denzin and Lincoln (2000) offer this definition:
Qualitative research is a situated activity that locates the observer in the world. It consists of a set of interpretative, material practices that make the world visible. These practices transform the world… qualitative
research involves an interpretative, naturalistic approach to the world.
(p.30)
A qualitative approach was chosen for the study, as it allowed the researcher to work in close association with participants. The study investigated how people think, learn or feel about events or instances. As this study explores the complex interconnection between people’s beliefs and practices and effects in the classroom, the methodology was chosen to “incorporate the existential experience of the participants themselves-their actions, thoughts, feelings, and perceptions- as a major focus for investigation and interpretation” (Frankel & Wallen, 1993). Working within the qualitative paradigm, I conducted in-depth case study (Cohen & Manion, 1997; Merriam, 1988; Yin, 1989) of master of education students. A case study as a research method “is a detailed examination of one setting, or a single subject [sic], or a single depository of documents, or one particular event” (Bogdan & Bicklen, 1998, p.54) and it enables
“intensive, holistic description and analysis of a bounded phenomenon” (Merriam, 1998, p. xiii).
Sampling involves decisions on the part of a researcher about how to use time during fieldwork, with whom to spend time, and whom to seek out (Ball, 1993). The sample also takes into consideration the practical aspects of the study; as Charles (1995) observes, “Where research is concerned with a population so large that it cannot be investigated in its totality, samples are a necessity” (p. 96). I employed purposeful sampling “to increase the utility of information to be obtained from small samples. The power and logic of purposeful sampling is that a few cases studied in depth yield many insights about the topic” (McMillan & Schumacher, 1997, p. 397). I selected 28 people for my study from the following four categories:
1. Seven students of the third cohort of master of education programme who were still comparatively new in this programme, and it is assumed that most of them are still in the process of learning about assessment. These students were selected to explore what their previous assessment practices were and what they learned about assessment in their initial three to four months of the two-year programme.
2. Seven students from the second cohort of master of education programme.
These students were nearing the completion of the programme, and soon they would go back to their own contexts and “face” the challenges of reality. They
were included to explore what their previous assessment practices were, what they learned at AKU-IED, and what they thought they would be doing in terms of assessment when they returned to their home institutions.
3. Students of the first cohort of master of education programme who already graduated from AKU-IED and had been working at their home institutions during my study. They were included so that in addition to what learned at AKU-IED, they could share what changes they tried to bring in their home institutions, in what ways they were successful or failed and why.
4. Seven faculty members of AKU-IED were also included as sample to explore their perceptions of how the students of master of education programmes learn assessment theories and practices at AKU-IED, and how they are prepared to bring changes in their home institutions.
The sample was expected to be sufficient to obtain a cross-section of views within each participant group. This representation is thought to give some comparison (though not very fair, as they are different individuals) across different cohorts according to their thinking about assessment and related aspects, when they first come to IED, then at the end of the programme and lastly after completing the programme.
Data were collected with the help of two tools mainly: interviews and document analysis. I used predominantly semi-structured interviews to interview all 28 participants individually (Hitchcock & Hughes, 1995; Cohen & Manion, 1997). Each participant was interviewed twice. Semi-structured interviews with in-depth probing questions enabled me to get rich, descriptive data in the participants’ own words. Hitchcock and Hughes (1995) observe that the semi-structured interview “tends to be most favoured by educational researchers since it allows depth to be achieved by providing the opportunity on the part of the interviewer to probe and expand the interviewee’s responses” (p. 83).
Audio recording of the interviews were done with the participants’ consent, to play the recorded material again for detailed analysis. As Frankel and Wallen (1993, p. 401) state
“The tapes may be replayed several times for continued study and analysis.”
Transcriptions of the recorded interviews were made. This was a time-consuming, but significant aspect of data collection. I found it very useful as I could get in depth commentary which was missed or skimmed over during the actual interview. While transcribing, I was mentally analyzing the findings. I was trying to arrange the findings accordingly, and also identify the themes that were coming out. After transcripts had been done, the copies were provided to the participants for content validity. The
researcher wanted the participants to have a look at the transcribed interview and confirm if it was transcribed properly, or some points were mistakenly put due to some technical or any other reasons. This was done to avoid misquotation. Some of the participants liked the idea of having a look at the things that he or she said during the interview. But there were cases when the participants changed many points after looking at copies again.
Either they found that they have said many things while being “too explicit” during the interview, or there were many points or “repetitions” that did not make sense according to them.
Document analysis was used as another tool for investigation. “Documentary analysis of educational files and records can prove to be an extremely valuable source of data.” (Johnson in Bell, 1993, p.67) The documents may be reports, minutes of meetings, memoranda, letters, files, reflective journals, or other kinds of written and non- written sources (e.g. photographs, recorded material). Among the documents that were reviewed were the Progress Reports developed at IED, AKU-IED Phase 2 Proposal, Students reflective journals, minutes of faculty meetings, memos prepared by the Associate Director (Academic), and others.
Data analysis is a rigorous continuous process of systematically searching and arranging the accumulated data (interview transcripts, field notes and memos, reflections, and other materials) in order to increase one’s understanding of them (Bogdan & Bicklen, 1998; Merriam, 1988; Niyozov, 2001). I used a combination of techniques – such as noting patterns and themes, testing plausibility, clustering, counting, making metaphors, making contrasts and comparisons, and noting relations between variables – to analyze the data and generate meaning from them (Miles & Huberman, 1994). As Denzin and Lincoln (1994) observe:
Qualitative research involves exhaustively examining the data first through careful reading and rereading of the data, keeping track of hunches, themes of interpretations and ideas, looking for emerging themes, constructing typologies, developing concepts and theoretical propositions, reading the literature, and developing a narrative line.
(cited in Tiede, 1996, p. 68)
Thus, data analysis involved making sense of the data by arranging them into coherent and plausible arguments. Even though I allocated time for post-fieldwork analysis, I began analysing the data from the moment I started collecting them and throughout the data collection; I continued during the writing-up as well.
Findings and Discussions
The findings and discussions are presented in three sub-sections that cover the M.Ed. students’ experiences prior to joining, at and after graduating from AKU-IED.
The focus will be on how the M.Ed. students experienced assessment practices prior and at AKU-IED, and what they hoped to implement upon returning to their home institutions.
Before AKU-IED
The study showed M.Ed. students generally bring together similar prior experiences with assessment, which can be characterized as ‘traditional’. In traditional assessment teaching and learning have been geared towards examinations and tests, thereby creating what the literature terms the ‘washback effect’. Examinations and (‘paper-and-pencil’) tests were often the only forms of conducting assessment. By and large assessment was conducted to check memorization of factual information and to monitor material coverage limited to the textbook or teachers’ knowledge. The forms of assessment generally promoted and required low-order thinking among students, with little if any emphasis on understanding. The type of assessment experienced promoted different kinds of malpractice such as cheating, leakage of question papers, taking bribes or other unfair means. The examination system was taken for granted rather than being questioned. Tension and anxiety appeared to accompany examinations, partly because examinations were not adequate to assess within three hours what had been learned in a whole year, thereby increasing stress.
The purposes of assessment were primarily to rank the students in class and across classes, to promote the students to the next class or to select them for further education. Students were/have been assessed individually but on a normative basis.
They viewed their peers as opponents with whom they had to compete. Competitive
marking negatively affected the classroom environment; that is, students did not help each other, and collaborative or cooperative learning was minimal or absent.
The environment of the assessment was characterized as very alienating and negative. There was little or no evidence that assessment was used to promote learning or bring about improvement. The following two incidents convey the participants’
emotional state while they were assessed.
A student one:
When I was a child I was a regular student, but due to some reason, perhaps I lost my exercise book, I had to complete my work within 3 or 4 days. I was late and when I approached teacher, she refused to check.
‘You are late!’, she shouted without listening to my explanation. Next time I saw her beating another student for being late [in submitting assignment]. So I did not have courage to go to her. What I did was I traced her signature, put it in my copy, and believe me, she... (perhaps she was in a hurry) she could not recognize and I was saved from punishment.
A student two:
He (teacher) used to make us stand first, and then he used to read the question and then he used to give us time to think. We were to stand so we could not write in our copies. Then he would say now: ‘Now it is time to sit, write down.’ While we were standing, nobody was allowed to touch a pen at all. ... I unconsciously touched my pen, as usually happens we want to play with something, and then he slapped hard me on my face. Since then I had developed a sort of fear of assessment.
To further illustrate the prior experiences of M.Ed. students prior to joining AKU-IED, a sample of comments is reported below. The responses are described in actual words or phrases, in order to make more accurate depiction of what the participants said. The key concepts regarding functions or purposes of assessment are grouped in italics in each case (See Table 1). However, they are not necessarily separate concepts and in many cases, they may overlap.
Table 1
Purposes of assessment Students’ voices
Promotion to the next class If someone missed the exam, he was not promoted, because the exam was the condition to get into the next class.
If I fail, I have to waste one year; I have to stay in the same class.
I was only focusing that examination would give a further step for promotion.
To my understanding the purpose of assessment was to pass or fail, to promote a person to another grade or retain him in his previous grade.
Ranking the students They used to label a person on the basis of their marks, if I got the highest marks then they would say ‘he is a very intelligent boy’.
My father was happy if I got first or second position, regardless of the percentage I got. I still remember when I got the first position in matric and I got a ‘hero’s welcome.
To categorize the students, who is the best, who is the second, etc. First, second, and third were considered as the most intelligent students of the class.
There were certain students who were labeled as failure.
For streaming To select students either for science group or the general group. Those who scored good marks in Science and Math, they were selected for Science group.
With examinations, they were branching, Arts and Science, going into A levels, going into college.
We had academic, modern, and commercial... Academics were all those students who had top ranks in Science and Math. Those who were good in the accounting, history and arts were put in the Commercial. And those who were the weaker ones, less able were put in the modern stream, and they were given a lot of typing skills, painting and art subjects.
Further education or jobs If you get good grades, you can get admission in good colleges.
If you got A’s then you could go for A levels... If you wanted to go on to university, you had to do this two year advance level in Science or in Arts. If you had B’s and C’s
you would not get a good chance to go to good universities.
We were concerned to succeed in the examination and get a job.
When the students were asked if there was any difference regarding assessment in their college or university experiences, most of them said that the pattern was the same, though intensity was more, and there was more work to do. They responded that everything was marked in the final examination. They questioned the validity of the examinations that set out to assess in two or three hours what was studied in one year.
Some participants said that they knew what questions would come in the exams, as there has been a pattern in the papers. A participant observed, “They [papers] come in alternative years. This year’s paper will not come again next year, but it may come after one year”.
Most students said that they did not have any formal training on how to assess the students. When they started teaching, they also assessed the way that they were themselves assessed, or the way in which the other teachers at school were assessing.
Sample faculty members responded to the questions about M.Ed. students’
experiences prior to AKU-IED with assessment in more or less similar ways that were described above. The people who facilitated the courses (sessions) were aware where the students were coming from. The questions were posed to get the faculty’s perspective and see the triangulation of data. One faculty member observed that “assessment is the passport, that is the ticket to next level, carrier”, while another faculty added, “They [students] all had been teachers and they had been doing the same assessment every year.
Their goal had been to make sure that the children pass their exams and they themselves became ingrained into drill”. Third faculty member mentioned that students came from a very competitive system, “Students come from systems where they have been competing... and the students have to work hard, and fight for marks.” He also mentioned that the paper-and-pencil type of assessment was predominant in the students’
experiences, where they were assessed mostly for memorization rather than for understanding or thinking. The functions were described as summative with little or no formative feedback. The other faculty member stated that the students “are mostly
coming from where they are exposed to right or wrong answers, and assessment is done at the end of a one whole year. There’s one small fragment of content where they just write the right answer.” She also mentioned that the students would be assessed mostly for recall or reproduction of information. She said, “The students have had experience of sitting in some kind of examination hall, trying desperately to recall as much as needed to be satisfied rather than really understand them. They got tuition to get a good mark in the examinations. It is a very artificial perception of assessment in terms of human potential and development.”
In conclusion, responses regarding the M.Ed. students’ experiences prior to AKU-IED, it is apparent that most of them had traditional assessment experiences. The students and faculty members, when interviewed, mentioned that the M.Ed. students usually hold very rigid traditional assessment beliefs due to their experience or exposure when they come to AKU-IED. The paper now turns to whether the students modify their concepts, in association with post graduate programme undertaken by AKU-IED.
Assessment at AKU-IED
On coming to AKU-IED, students generally expected that a different approach would be used to assessment, although they were usually uncertain about what these approaches would be. At AKU-IED, the students learned the new forms of assessment from class sessions and topics studied. Equally importantly, the students learned about assessment both formally and informally from the ways in which they are assessed by their instructors. There is a multifaceted way of assessing the M.Ed. students. There is no reliance on summative forms of assessment, such as major paper-and-pencil tests.
Rather, there is a range of strategies used in assessment. These include: written assignments, group work, practical mini-activities, class participation, reflective journals- self assessment, peer assessment, and presentations.
The students’ initial reactions to assessment practices at AKU-IED varied.
There were such students who were positively surprised when they learned that they would be assessed on an on-going basis, rather than a summative or examination basis.
For example, one student commented, “I felt very relaxed, because there was not an examination, like a three hour or a one hour examination, assessment was only on a daily
basis, attendance, participation among the groups and presentations”. However, there was another participant with quite opposite feelings towards various aspects of the assessment practice at AKU-IED. She did not feel relaxed with some of the practical activities at AKU-IED, such as micro-teaching, as she “was conditioned to produce a written paper.” Consequently, she felt pressure while doing practical activities that had never been assessed before in her experience.
Two people, who expected that English would be valued, were disillusioned that not English, but “ideas were valued at AKU-IED”. These students had a particularly good command of English and generally secured top positions.
Some students commented on the significance of feedback on their learning from assessment. They said that feedback given by the tutors was extremely useful. This was identified as one of the things that they would like to take with them to their classes in their home contexts, because feedback served the formative purpose of providing very good information to students.
Reconceptualization of assessment practices
When asked if they had reconceptualised their assessment thinking at AKU- IED, there was a range of responses. For example, these varied from “I came already reconceptualised here” to “I cannot say I changed, I say that I reinforced my ideas” to
“there was a total change”, or “I started thinking differently”. Some participants said that they definitely changed their thinking on assessment at AKU-IED. Most said that previously they would not even question the existing examination system, but now they do. Another participant mentioned that prior to coming to AKU-IED, their concern was mainly technical - how to improve tests and examination papers, how to deal with malpractice that happens during assessment - rather than thinking of philosophical aspects and questions related to purposes and methods of assessment. Below is a cross- section of views of students on their reconceptualization.
It is not good that we are just evaluating students just for three hours.... But there should be on going assessment. While students are learning, why are we not monitoring, why we are not recording their performance in the class?
We ignore many aspects; we assess just cognitive aspects of the assessment. We never assess skills and attitudes. There are many other things in development of the personality of a child. Now I have started thinking that our assessment should be broadened, not be so focused to one aspect. It should be holistic.
Now I think that individual has to be compared with his or her own progress, but not with other people. Assessment is done with people and for people rather than at and to people. I think that is just trying to show others where they are at is not what we want.
It should be trying to facilitate their academic learning.
To summarize, there is evidence of a shift in students’ perceptions of assessment, although the extent differs from person to person. For most, the learning experience created a paradigm shift in their thinking from traditional assessment to alternative ways. Irrespective of whether the students claimed or appeared to have undergone a paradigm shift, all of them indicated either directly or through their comments that they had re-thought assessment procedures and concepts such as validity, reliability, bias, purposes, types, and side-effects. The students said they had reconceptualised or learned about assessment primarily in the following ways: modes of assessment by their instructors, sessions on assessment and on education in general, reading literature related to the topic, discussions with other people, and visiting schools during modules.
In the data previously reported, the focus has been on assessment. However, reconceptualization of assessment does not take place in a vacuum in relation to other experiences undergone. A relationship can be expected between re-thinking of assessment and re-thinking of teaching and learning, possibly extending to other areas of a person’s philosophy as well. Some comments below from students illustrate this broader dimension of reconceptualization. While not necessarily stated as forcefully, similar themes or observations were apparent in a number of interviews during this study.
Student 1
You see, the journey has been very tedious and dangerous from one side of beliefs to another side. These were two extremes, I was on the one side and then I was taken around to another side... You see I
came from a place where people do not value ideas, but they value status...
Student 2
AKU-IED's culture is quite unique... number one they [people] respect each other, number two they value ideas, number three they are open to suggestions, number four there is expressed that all are equal. You call even the director by his first name. The context where I was coming from I could not stand let alone sit in front of the director. If director is coming I am supposed to stand up, even if I am working. I have to leave all my work and stand up in respect of his highness, ‘the director is coming’ [author’s emphasis]...
Student 3
... the other perception built was that I would not say that I was religious person before coming to AKU-IED, but I became secular here. I started to think that everybody has a right to live. Even if he or she belongs to a different religion. I stopped discriminating people on the basis of religion, cast or color…. Whether you are a follower of Hinduism, and you worship idols, Buddhist, or a Jew, or a Christian or whatever! That was a great change... Everybody has faith, right to say and have ideas and his or her ideas should be respected.
To sum up the students who come to AKU-IED by and large have similar experiences, and their expectations of the AKU-IED's assessment, reactions to it, and the thoughts on reconceptualization form to draw a similar pattern, that people usually come more or less from a ‘traditional past’ and ‘accept willingly and with pleasure’ the ideas introduced at AKU-IED. During the interviews some participants commented about reconceptualization whether it was a big shift or change from one paradigm to another or whether it was a mere learning more about a concept.
Espoused and actual assessment practices
One of the assumptions which was made, at least implicitly, in setting up this study was that students would enter AKU-IED with an experience based on one assessment paradigm and would adopt an alternative approach while at AKU-IED.
Further, it was expected that students’ experience of tension or dissonance between their
‘new’ views and ‘system’ views would be experienced, but only after leaving AKU-IED and returning to their home work contexts. In fact, an unexpected but important finding has been that students experienced dissonance while at AKU-IED. Rather ironically, this arose from an apparent discrepancy between what students felt were AKU-IED's values on assessment versus AKU-IED's practice on assessment. This issue, which raised major questions and concerns for many students, is discussed at some length in the section below.
The data from the participants reveal that reconceptualization at AKU-IED is considerably more complex than has been assumed or outlined above. This complexity relates to the apparent or perceived discrepancy between positions advocated at AKU- IED and the actual assessment practice. This issue is elaborated later in the chapter. The director of AKU-IED summarized the discrepancy as follows.
Many students had endorsed other than traditional ways of assessment because of AKU-IED’s impact. However, in some respects, AKU- IED’s actual practice in assessing the M.Ed. students seemed to reflect more traditional assessment assumptions than its own teaching recommended or advocated. Consequently, in a number of cases the students reverted to behaviors and attitudes associated with traditional assessment, including those students who accepted the notions of alternative assessment in principle or theory. As a result, the AKU- IED has been experiencing a paradox in which espoused beliefs contrast with practical realities.
It has already been pointed out earlier that M.Ed. students came from an educational background in which their assessment experience could be characterized as traditional. It has been shown that students experienced an alternative approach to assessment at AKU-IED and, with few exceptions, most students adapted to, and
adopted, the alternative approach willingly and without significant difficulty. Therefore, students experienced relatively little ‘dissonance’ in shifting from their prior assessment stance to an alternative one which places more emphasis on continuing and participatory strategies.
However, an unexpected issue arose during the M.Ed. course and, as a result of which, many students appeared to have been placed in a situation of dissonance. In essence, they needed to resolve the unanticipated dilemma of having, on the one hand, adopted a new approach to assessment which was apparently endorsed and valued by the institution while, on the other hand, finding that some of the assessment grading practices at this institution appeared to reflect principles more associated with traditional forms of assessment (such as normative, summative, and competitive). In fact, the dilemma created by the institution’s practice probably illustrates a much more widespread phenomenon in higher education, namely the need for universities to respond to the pressure to grade and select and compare students for a range of accountability and quality assurance reasons. Specifically, most higher education institutions either choose or are required to allocate summative grades on a normative assumption. Typically, such grades will be in the form of a grade point average, or a letter grade - A, B, C, etc., or a percentage or other quantitative indicator. As AKU-IED is constrained by the procedures of the University articulated and monitored by the Board of Graduate Studies, AKU-IED, given its commitment to alternative approaches to teaching and learning, and appropriate assessment practice, there may be little option but to espouse one practice while implementing another. The result of this apparent discrepancy was quite dramatic as illustrated through data presented below.
The impact of grading
Nearly all the participants pointed to unusual climate of AKU-IED, which is very friendly, and supportive. People respect each other and their ideas. This supportive environment helped the students to act accordingly. There has been a lot of learning especially affective domain of learning because of being endorsed to the environment.
However, majority of the participants said that the behaviors of people considerably changed negatively after the grading policy was announced to them. As two participants said the ‘collegiality has died after the announcement of grading policy.’ They said that
the people stopped being cooperative, no one was helping others, everyone worried about themselves. There appeared some atypical behavior as hiding material from others access, academic cheating, plagiarism, competition and jealousy. The students started to perform well to get extrinsic reward, which became pre-dominant to intrinsic purposes of learning. AKU-IED’s grading practice has caused a lot of disturbances among the M.Ed.
students. The students reported that they were quite unhappy that reporting of grades was handled insensitively, which caused a lot of disturbance. The grade results were posted in the classroom, which was insensitive for many people. People started comparing and contrasting their grades, and their expectations of a person, which led to further complications of the situation. Some students “...felt ridiculed or insulted for getting lower grades than what they had been expecting” according to one student. As different people reacted to the grade issue in various ways, their reactions and view-points are reported verbatim and at some length.
When we were given grades some people became very upset. Some people said that tutors were biased against them and there was a lot of commotion. So it was decided among the faculty members that no grades will be given. There were so many ill-feelings among the faculty and among the CPs. CPs were blaming faculty, faculty were blaming CPs. It was not a good experience.
When we got grades, I think the people become frustrated. Division has started.
Division means categories. Some people who got A they are in A category, who got B are in B category, some were in C. So it was observed and felt that this grading is affecting collegial attitude. And it is not good for cooperation, for cooperative learning.
If a person has very high expectations of... is not very analytical of her or his own learning and has very high expectations and is not assessed as highly that could have a very demoralizing / demotivating effect.
At the time of assignment I think the colleagues would usually hide, cover page of the books, and not help each other.
...everybody wants to get A, because it gives feeling of happiness. It gives that you have been rewarded for your hard work. It gives incentives to do more. In
the final assessment you will be evaluated and if you have got A or B throughout your studies in AKU-IED, you will be considered a good person and in future work of AKU-IED they require such a person, then they will say we have got this person who was very consistent.
Some people were lowly graded, and some were highly graded. To me if a person fails that grade, the equal responsibility lies with faculty also. Because if a person gets A then the faculty also has a role.
My principal asked me what grades you brought. People treat you according to the grades. I am going to be assessed on the grades that I will bring.
I don’t know if AKU-IED can do away with grades. I feel that what they have said if you get four As and in dissertation A, then you will get a distinction.
They are going against policy. They say that grade does not matter. But how can you say if you get an A you will get a distinction, and don’t think about grades? It is a mismatch.
In one of our modules a lot of us got A grades. Then director came and said it was not the right thing. “The tutor who marked was not really paying attention.”
They said that it is wrong, it should be bell curve... they were not happy why everybody was getting an A.
Faculty members were the people who are involved in the above-mentioned process. They did a lot of hard work regarding planning, instruction and assessing. The following statements are the faculty members’ views, reactions, and observations of the grade related issues in M.Ed. 3 and the other two cohorts as well.
What happened in our first M.Ed. programme, we shared assessment policy with the students and we had been doing things, but unfortunately in the middle of the programme there was a problem. People got some frustration with the assessment marks. We thought that since AKU-IED's purpose is to flourish, nurture the environment of the collaboration and collegiality. We thought perhaps collaboration and collegiality are going to be defeated. So we made a decision and stopped distributing grades.
In summary grading which caused many troubles at AKU-IED to students and teachers lately. It is apparent that this is also because of the students prior to AKU-IED experiences where the academic achievement by any means, fair or not fair, which directly reflect and influence the human behavior in a new environment, and also sense or awareness of ‘life’ after AKU-IED. When people would be facing again the same system which would seek from them ‘degrees’ not ‘ability’. A letter grade would tell more about a person than anything else. And in the competitive world, where the resources, academic privileges are limited one would really go against his or her principles and act according to the demands of the situation. Competition is reported to exist in different forms between the students to get higher grades.
After AKU-IED
So far, data have been reported on students’ assessment experience prior to AKU-IED and their reactions to assessment at AKU-IED. A complex picture has emerged, particularly in relation to the espoused / practiced dimension. The final area investigated in this study concerns the actual or anticipated assessment practices after leaving AKU-IED. The M.Ed. graduates take on positions of responsibility after leaving, with a particular mandate to improve quality of teaching and learning. Therefore, they encounter many situations in which assessment practices and concepts need to be addressed and / or challenged. However, these take place in a system which practices and values traditional, summative forms of assessment. The question then arises as to what barriers or constraints the M.Ed. graduates face and how they address these. The students, who, as illustrated previously, have adopted new ideas of assessing at AKU- IED, also stated that they would not be assessing their students in the way they did prior to AKU-IED. However, it was also noted that much would depend on the context where they are going to after competing the programme. The students are, in general, aware of barriers or constraints that they have to encounter in their future work. Those barriers can be illustrated with the following quotations (Table 2).
Table 2: Hurdles to assessment reforms
School administrators They will expect immediate outcomes, and significant progress in the results in examinations would bring fame, reputation to the school.
Colleagues Many teachers are still working in an old pattern, and they would require strong alternatives if they are to change their traditional concepts.
Difficult to change the teachers, to equip them with the skills required for new ways of assessment.
The teachers help their students in achieving high grades, because based on those high grades those teachers were expecting high performance allowance [financial incentives].
School facilities and time School facilities will restrict the use of many innovative ways of assessing, for example, group work would be difficult to organize and assess in rooms which are small, with heavy immobile desks arranged in rows.
Time is a crucial factor, as the school has its own agenda.
Teachers have to complete the syllabus, and prepare the students for the exams, which makes it extremely difficult to assess informally.
This [large class] is a problem of developing country. We haven’t resources, and due to lack of resources we cannot get the same standard.
It is difficult to teach and to assess the students within 30 minutes. I think it is not only difficult, it is impossible.
There are about 100 students in a class. I want to address with this issue with my school system. I would be asking the students to work in groups, as I think they could learn more in groups.
If there are more than 60 students, I may not be able to assess every student, to have record of on-going assessment.
Parents For many parents, the learning of their children can be seen from the amount of the written work a student does in the class, and memorization or maybe understanding of the knowledge which is given in the textbook.
Parents would be expecting their children do perform well
in the examinations and be among the students who get top positions.
Board / National Examinations
Board examinations usually require lower order thinking and the questions are set in a pattern that repeats every second year. There would be little sense to teach the students in new ways, but to assess in old ways. This way would lead to negative impact of examination into teaching and learning process. Indeed that is happening in many classroom practices that school are examination-driven.
Students will have to memorize to pass Board exams. I think it will create a bad effect on students. If the students are taught for understanding, if there’s on-going assessment, but for Board Exams they have to write what is written in the book. It will be difficult thing for the students also to shift from one to another assessment practice.
Board examination or National examination in some countries, are the major barriers for the teachers. It is so exam-oriented... we cannot go against the government decision. National exam is there, because they select people for jobs, and for universities.
... right now there are certain teachers who are very good and using different ways of teaching in lower classes, but when they come to classes 9 and 10 they don’t use those methods because of Board exams.
Malpractice
(Cheating, leakage of papers, setting papers, etc.)
The students copy during exams. Some teachers are also involved in cheating, they request for marks to other teachers.
The students do understand that when they return to their home institutions they need to work together with other people in order to make improvements in the assessment system. The participants indicated that changes in the assessment system would not take place in a ‘vacuum’, other aspects of teaching and learning must change. He said,
“Working in school would mean that other things also change side by side, you just can’t change one assessment. We would like to work in collaboration with other colleagues to change the assessment ways at least within our reach.” Faculty members recognize the dilemma illustrated above, for example, one of them said:
There is a danger that the graduates might revert to their old ways of assessment.
Those people who end up teaching higher classes, they might go back whether they like it or not. If they are teaching class which has got Board Examination, surely they will have to revert. In Pakistani schools, tests mean a lot. They can’t change the school overnight. When you are trying to change the school, the school is also trying to change them.
Much of the discussion in this paper could be constructed as rather pessimistic.
To some extent, this may be due to the focus question relating to ‘barriers’ which would need to be addressed to enable positive change. To some extent, the pessimism reflects structural and political realities. However, there is considerable evidence of positive attitudes and outcomes, and a commitment to bringing about school improvement.
Clearly, the challenge is to maintain this commitment in the face of significant constraints. Recognition of the constraints is perhaps a major first step. The views of some faculty members give some cause for affirmation.
The students would go home and look very critically at the assessment policies for their subject area in their contexts and take on leadership role to change many of the highly restrictive oppressive policies that are usually centered in exams that are laid at the government level for all students. I really would want the teacher to be an agent of change, not only at the school level or classroom level, but to feel empowered.
Implications
There are several implications driven from this study. The assessment concept does not have the centrality which it ought to have at an institution which is trying to make qualitative change and improvement in education. Unless and until assessment is brought to the forefront we cannot expect genuine progress in the overall quality of teaching and learning, as the negative washback effect of assessment will act against the efforts, time and resources expended to bring about improvement. How can education make a real difference when the system must respond to the pressures generated by examinations and other characteristics of traditional assessment? It is important to improve curriculum development, instruction and classroom interactions, but much of the
work maybe meaningless until serious measures are taken regarding improvement of assessment. How much difference can teacher education make when we still have dissonance between espoused and actual assessment practice at an institution which aims to promote ‘paradigm shifts’?
The ‘assessment dilemma’ impacts on students’ behavior, attitudes, and relationships with each other as well. A comprehensive study into this dilemma with respect to assessment may be justified. A research study can be done into how to address this issue of dissonance between espoused and actual assessment practice and illustrate ways of bridging the gap between them. Innovative ways of linking traditional practice in a constructive manner with new ideas of assessment (self-assessment, peer assessment, assessment of group-work, profile or portfolio) should be explored and investigated.
AKU-IED is one of the few institutions, if not the only one in this context, that is attempting to reconceptualise assessment notions. In fact, AKU-IED itself is a victim of the existing system of education and therefore lacks flexibility. The broader issues must be addressed in university-wide policy, not only at this University but also in higher education generally. It is recommended that a case study be conducted, or at least information be obtained on, the experience of higher education institutions elsewhere in the world which have tackled the problems and issues illustrated in the present study.
A further implication from this study is that AKU-IED needs to look more seriously at faculty development in the area of assessment. It seems essential to ensure that faculty understand the full range of assessment strategies and options, their implications and the issues which arise from their use.
With respect to the educational system at large, AKU-IED might organize educational issues sessions or seminars on assessment, focusing on dilemmas and possibilities. Various stakeholders (heads of schools, District Education Officers, university and school teachers, parents, etc.) could be invited for panel discussions. More work can be done at a policy makers’ level, including not only bureaucrats but also politicians. Perhaps, AKU-IED should liaise closely and negotiate with Boards of Examinations, because unless there is a change in the requirements of Boards there is little hope that the situation can improve at school level. As some of the participants stated, changes may occur in lower classes in schools, but in secondary, and especially
classes that have public examinations, not much change can be envisaged. Board Offices have to be addressed as potential partners to improvement, not as ‘stone-age obstacles’.
Literature review of the relevant materials has been helpful to establish framework for the study. What is increasingly apparent from the review is that a list of descriptions and characteristics of dichotomous assessment approaches, for example traditional and alternative, formal and informal which are given in my literature. There is very little literature on how the shift from more traditional to alternative approaches to assessment can take place, especially in a developing country context. Thus, more study should be done to look beyond descriptions and characteristics to the ways of actually achieving objectives and aims, that so many people endorse and advocate.
Conclusion
To conclude I turn the discussion to social issues that arise out of education, and in this case assessment. The following arguments are based on my values in conjunction with the insights and questions emerging from the present study. I strongly believe that assessment can serve to help break social reproduction, to create a society with more justice for those who are underprivileged and disadvantaged. Many people speak for elimination of tests, implying that elimination of tests will automatically solve many problems of inequality. However, Scarr (cited in Gipps & Murphy, 1996, p.15) argues that in the “absence of tests other less objective assessments will be found and used;
abandoning tests cannot make society more pluralistic, ensure equal rights or redistribute social and economic benefits”. We all can strive to make assessment fair and just for all groups, thus pursuing equity as a principle, along with a belief that change is possible.
References
Airasian, P. (2005). Classroom assessment: concepts and applications. Boston: McGraw Hill.
Capper, J. W. (1996). Testing to learn - learning to test. Washington: Academy for Education Development.
Farooq, R. (1994). Education system in Pakistan: issues and problems. Islamabad: Asia Society for Promotion of Innovation.
Fisher, P. (1991). Recording achieving in primary schools. Oxford: Basil Blackwell Ltd.
Gipps, C. & Murphy, P. (1996). A fair test: assessment, achievement and aquity.
Buckingham: Open University Press.
Gipps, C. (1990). Assessment: a teacher’s’ guide to issues. London: Hodder &
Stoughton.
Goodwin, A. L. (1997). Assessment for equity and inclusion: embracing all our children.
New York: Routledge.
Harris, M. & McCann, P. (1994). Assessment. Oxford: Heinemann.
Kanu, Y. (1994). Research. (a classroom handout). Karachi: The Aga Khan University.
Institute for Educational Development.
Kellaghan, T. & Greaney, V. (1992). Using examination toimprove education: a study in fourteen african countries. Washington, D. C. The World Bank.
Linn, R. & Miller, D. (2005). Measurement and assessment in teaching. Columbus, Ohio: Pearson.
Mackintosh, H. G. & Hale, D. E. (1976). Assessment and the secondary school teaching.
London: Routledge and Keagan Paul.
McMillan, J.H. (1997) Classroom assessment: principles and practice for effective instruction. Boston: Pearson.
Murphy, R. & Torrance, H. (1991). The changing face of educational assessment.
Philadelphia: Open University Press.
Phase 2 Proposal, AKU-IED. 1997. Karachi.
Popham. W.J. (2001). The truth about testing: an educator’s call to action. Alexandria, VA. USA: ASCD.
Progress Report. 1995-1997. AKU-IED, Karachi.
Singh, R. (Ed). (1993). Indian education. Delhi: Commonwealth Publishers.
Sinha, A. (Ed.). (1993). Educational challenge in a dynamic world. Education and Development. Vol. 1. Delhi: Sarup & Sons.
Torrance, H. (1995). Evaluating authentic assessment. Buckingham: OUP.
Correspondence
Duishon Alievich Shamatov, Ph.D Email: [email protected]