%PDF-1.3 % Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment. 1. This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway. HEFCE developed an initial methodology that was then tested through a pilot exercise. It is now possible to use data-mining tools to extract specific data from narratives or unstructured data (Mugabushaka and Papazoglou 2012). stream 0000334705 00000 n Assessment refers to the process of collecting information that reflects the performance of a student, school, classroom, or an academic system based on a set of standards, learning criteria, or curricula. In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. A collation of several indicators of impact may be enough to convince that an impact has taken place. If knowledge exchange events could be captured, for example, electronically as they occur or automatically if flagged from an electronic calendar or a diary, then far more of these events could be recorded with relative ease. We take a more focused look at the impact component of the UK Research Excellence Framework taking place in 2014 and some of the challenges to evaluating impact and the role that systems might play in the future for capturing the links between research and impact and the requirements we have for these systems. When considering the impact that is generated as a result of research, a number of authors and government recommendations have advised that a clear definition of impact is required (Duryea, Hochman, and Parfitt 2007; Grant et al. 0000007559 00000 n The development of tools and systems for assisting with impact evaluation would be very valuable. Worth refers to extrinsic value to those outside the . HEIs overview. The University and College Union (University and College Union 2011) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. evaluation of these different kinds of evaluands. 0000004731 00000 n 0000346296 00000 n This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. Attempting to evaluate impact to justify expenditure, showcase our work, and inform future funding decisions will only prove to be a valuable use of time and resources if we can take measures to ensure that assessment attempts will not ultimately have a negative influence on the impact of our research. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable. There are areas of basic research where the impacts are so far removed from the research or are impractical to demonstrate; in these cases, it might be prudent to accept the limitations of impact assessment, and provide the potential for exclusion in appropriate circumstances. What indicators, evidence, and impacts need to be captured within developing systems. (2007), Nason et al. The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, Measuring Impact Under CERIF at Goldsmiths, Anti-Impact Campaigns Poster Boy Sticks up for the Ivory Tower. 0000011585 00000 n 0000002868 00000 n 0000002318 00000 n Narratives can be used to describe impact; the use of narratives enables a story to be told and the impact to be placed in context and can make good use of qualitative information. There is a distinction between academic impact understood as the intellectual contribution to ones field of study within academia and external socio-economic impact beyond academia. Introduction, what is meant by impact? It is perhaps assumed here that a positive or beneficial effect will be considered as an impact but what about changes that are perceived to be negative? %PDF-1.4 % Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. Merit refers to the intrinsic value of a program, for example, how effective it is in meeting the needs those it is intended help. The basic purpose of both measurement assessment and evaluation is to determine the needs of all the learners. Hb```f``e`c`Tgf@ aV(G Ldw0p)}c4Amff0`U.q$*6mS,T",?*+DutQZ&vO T4]2rBWrL.7bs/lcx&-SbiDEQ&. It incorporates both academic outputs and wider societal benefits (Donovan and Hanney 2011) to assess outcomes of health sciences research. While looking forward, we will be able to reduce this problem in the future, identifying, capturing, and storing the evidence in such a way that it can be used in the decades to come is a difficulty that we will need to tackle. Prague, Czech Republic, Health ResearchMaking an Impact. Although metrics can provide evidence of quantitative changes or impacts from our research, they are unable to adequately provide evidence of the qualitative impacts that take place and hence are not suitable for all of the impact we will encounter. The transfer of information electronically can be traced and reviewed to provide data on where and to whom research findings are going. Throughout history, the activities of a university have been to provide both education and research, but the fundamental purpose of a university was perhaps described in the writings of mathematician and philosopher Alfred North Whitehead (1929). SIAMPI has been used within the Netherlands Institute for health Services Research (SIAMPI n.d.). By allowing impact to be placed in context, we answer the so what? question that can result from quantitative data analyses, but is there a risk that the full picture may not be presented to demonstrate impact in a positive light? What emerged on testing the MICE taxonomy (Cooke and Nadim 2011), by mapping impacts from case studies, was that detailed categorization of impact was found to be too prescriptive. The ability to write a persuasive well-evidenced case study may influence the assessment of impact. Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact? 0000342798 00000 n While valuing and supporting knowledge exchange is important, SIAMPI perhaps takes this a step further in enabling these exchange events to be captured and analysed. Differences between these two assessments include the removal of indicators of esteem and the addition of assessment of socio-economic research impact. As a result, numerous and widely varying models and frameworks for assessing impact exist. Definition of evaluation. (2005), Wooding et al. Cooke and Nadim (2011) also noted that using a linear-style taxonomy did not reflect the complex networks of impacts that are generally found. For example, the development of a spin out can take place in a very short period, whereas it took around 30 years from the discovery of DNA before technology was developed to enable DNA fingerprinting. The traditional form of evaluation of university research in the UK was based on measuring academic impact and quality through a process of peer review (Grant 2006). HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). This is particularly recognized in the development of new government policy where findings can influence policy debate and policy change, without recognition of the contributing research (Davies et al. 0000009507 00000 n Many theorists, authors, research scholars, and practitioners have defined performance appraisal in a wide variety of ways. Figure 2 demonstrates the information that systems will need to capture and link. From 2014, research within UK universities and institutions will be assessed through the REF; this will replace the Research Assessment Exercise, which has been used to assess UK research since the 1980s. Perhaps the most extended definition of evaluation has been supplied by C.E.Beeby (1977). Organizations may be interested in reviewing and assessing research impact for one or more of the aforementioned purposes and this will influence the way in which evaluation is approached. As such research outputs, for example, knowledge generated and publications, can be translated into outcomes, for example, new products and services, and impacts or added value (Duryea et al. Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. The RQF pioneered the case study approach to assessing research impact; however, with a change in government in 2007, this framework was never implemented in Australia, although it has since been taken up and adapted for the UK REF. At least, this is the function which it should perform for society. The case study does present evidence from a particular perspective and may need to be adapted for use with different stakeholders. Its objective is to evaluate programs, improve program effectiveness, and influence programming decisions. To evaluate impact, case studies were interrogated and verifiable indicators assessed to determine whether research had led to reciprocal engagement, adoption of research findings, or public value. Researchers were asked to evidence the economic, societal, environmental, and cultural impact of their research within broad categories, which were then verified by an expert panel (Duryea et al. The Goldsmith report (Cooke and Nadim 2011) recommended making indicators value free, enabling the value or quality to be established in an impact descriptor that could be assessed by expert panels. Two areas of research impact health and biomedical sciences and the social sciences have received particular attention in the literature by comparison with, for example, the arts. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. Understand. 0000011201 00000 n "Evaluation is a process of judging the value of something by certain appraisal." Characteristics of evaluation in Education Below are some of the characteristics of evaluation in education, Continuous Process Comprehensive Child-Centered Cooperative Process Common Practice Teaching Methods Multiple Aspects Continuous Process Any tool for impact evaluation needs to be flexible, such that it enables access to impact data for a variety of purposes (Scoble et al. Concerns over how to attribute impacts have been raised many times (The Allen Consulting Group 2005; Duryea et al. In 200910, the REF team conducted a pilot study for the REF involving 29 institutions, submitting case studies to one of five units of assessment (in clinical medicine, physics, earth systems and environmental sciences, social work and social policy, and English language and literature) (REF2014 2010). 2009; Russell Group 2009). Research findings including outputs (e.g., presentations and publications), Communications and interactions with stakeholders and the wider public (emails, visits, workshops, media publicity, etc), Feedback from stakeholders and communication summaries (e.g., testimonials and altmetrics), Research developments (based on stakeholder input and discussions), Outcomes (e.g., commercial and cultural, citations), Impacts (changes, e.g., behavioural and economic). The Payback Framework enables health and medical research and impact to be linked and the process by which impact occurs to be traced. A very different approach known as Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions (SIAMPI) was developed from the Dutch project Evaluating Research in Context and has a central theme of capturing productive interactions between researchers and stakeholders by analysing the networks that evolve during research programmes (Spaapen and Drooge, 2011; Spaapen et al. 0000010499 00000 n Different authors have different notions of educational evaluation. In education, the term assessment refers to the wide variety of methods or tools that educators use to evaluate, measure, and document the academic readiness, learning progress, skill acquisition, or educational needs of students. The most appropriate type of evaluation will vary according to the stakeholder whom we are wishing to inform. The Oxford English Dictionary defines impact as a Marked effect or influence, this is clearly a very broad definition. 0000012122 00000 n The definition of health is not just a theoretical issue, because it has many implications for practice, policy, and health services. By asking academics to consider the impact of the research they undertake and by reviewing and funding them accordingly, the result may be to compromise research by steering it away from the imaginative and creative quest for knowledge. different things to different people, and it is primarily a function of the application, as will be seen in the following. Impact has become the term of choice in the UK for research influence beyond academia. In the majority of cases, a number of types of evidence will be required to provide an overview of impact. Donovan (2011) asserts that there should be no disincentive for conducting basic research. They risk being monetized or converted into a lowest common denominator in an attempt to compare the cost of a new theatre against that of a hospital. Recommendations from the REF pilot were that the panel should be able to extend the time frame where appropriate; this, however, poses difficult decisions when submitting a case study to the REF as to what the view of the panel will be and whether if deemed inappropriate this will render the case study unclassified. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide, This PDF is available to Subscribers Only. This report, prepared by one of the evaluation team members (Richard Flaman), presents a non-exhaustive review definitions of primarily decentralization, and to a lesser extent decentralization as linked to local governance. To achieve compatible systems, a shared language is required. The range and diversity of frameworks developed reflect the variation in purpose of evaluation including the stakeholders for whom the assessment takes place, along with the type of impact and evidence anticipated. This involves gathering and interpreting information about student level of attainment of learning goals., 2. Overview of the types of information that systems need to capture and link. As Donovan (2011) comments, Impact is a strong weapon for making an evidence based case to governments for enhanced research support. In the UK, there have been several Jisc-funded projects in recent years to develop systems capable of storing research information, for example, MICE (Measuring Impacts Under CERIF), UK Research Information Shared Service, and Integrated Research Input and Output System, all based on the CERIF standard. It is a process that involves careful gathering and evaluating of data on the actions, features, and consequences of a program. What is the Concept and Importance of Continuous and Comprehensive Evaluation. Oxford University Press is a department of the University of Oxford. 0000001087 00000 n Gathering evidence of the links between research and impact is not only a challenge where that evidence is lacking. In demonstrating research impact, we can provide accountability upwards to funders and downwards to users on a project and strategic basis (Kelly and McNicoll 2011). This might include the citation of a piece of research in policy documents or reference to a piece of research being cited within the media. Where quantitative data were available, for example, audience numbers or book sales, these numbers rarely reflected the degree of impact, as no context or baseline was available. Every piece of research results in a unique tapestry of impact and despite the MICE taxonomy having more than 100 indicators, it was found that these did not suffice. The Payback Framework is possibly the most widely used and adapted model for impact assessment (Wooding et al. They aim to enable the instructors to determine how much the learners have understood what the teacher has taught in the class and how much they can apply the knowledge of what has been taught in the class as well. Accountability.
Do Catholic School Teachers Get A Pension,
Hail Hail State Police Cadence,
Chris Walker Adairville,
Police News In Murray Bridge,
Roommate Harassment Laws California,
Articles D