Improving how we evaluate research: how we’re implementing DORA

At CRUK, we spend more than £380 million of our donors’ money on research each year. We work hard to make sure we fund the highest quality projects which will have the greatest possible impact on beating cancer, and we’ve been improving how we assess research applications.

As signatories to the Declaration on Research Assessment (DORA), we’re committed to assessing the quality and impact of scientific research through means other than journal impact factors, and we’ve implemented several initiatives over the past year that align with DORA principles. 

We caught up with Dr Fiona Reddington, Head of Population, Prevention and Behavioural Research at CRUK, who also sits on the DORA steering committee, to find out more about what’s been going on at CRUK and in the wider research community.  

Why does CRUK care about improving the way we evaluate research?

We’re committed to seeing the day when all cancers are cured, and we set an ambitious target of seeing three in four patients with cancer surviving their disease by 2034. To reach this goal, we want to fund internationally competitive research that is of the highest quality, and support outstanding cancer scientists as they transition to independence. A central component feeding into our funding decisions is how we assess the quality of the research that is presented to us by applicants; this can determine funding, promotion, hiring and tenure decisions, so it is key that we get it right.

Historically, journal impact factors have been used as a surrogate to judge individual researchers; this seems rather counter-intuitive when considering how inexact the relationship is, and here at CRUK we recognise the need to change this. Since signing up to DORA, we’ve been making headway implementing changes to our application process and criteria for judging researcher quality. We stress that while journal articles are important, we value and reward all types of research outputs. We ensure applicants have an appropriate arena to highlight their key research achievements, and that our reviewers are well-aware of recognising and evaluating this data to make informed funding decisions.

What are some of the highlights from the past year?

We’ve been busy amending our application guidelines in line with DORA principles, and we have been working hard to ensure our funding processes are aligned with best practice. One of the first changes we made was to start asking applicants specifically about their ‘research outputs’ more broadly, rather than just their publication record.

Another noteworthy change is the introduction of a new flexible research careers policy which supports varied and flexible career paths. We recognise the need to be flexible with eligibility criteria for fellowship applicants, particularly after periods of long-term leave, and we are very supportive of researchers applying and working part-time or flexibly on any of our awards. Our new policy reflects this by allowing amendments to active grants, and any periods of leave to be factored when assessing eligibility for grants and any subsequent funding decisions.

How have you equipped your peer reviewers and committee members to ensure the principles of DORA are implemented during the funding process?

We’ve developed a fair, robust and transparent system to evaluate applications for funding. All our grants are awarded in open competition and they are reviewed by independent experts in the research community. We now remind our peer reviewers and committee members of DORA principles throughout the funding process, and we’ve also amended our guidelines for applicants to reflect this.

When considering the track record of researchers and their suitability in delivering the proposed research, our reviewers are advised to consider the value and impact of all research outputs, including preprints, training, contribution to consortia, patents, and sharing of key datasets, software, novel assays and reagents, in addition to research publications. When considering research publications, our reviewers are reminded that the content of a scientific paper and its influence in the field holds more significance than publication metrics or where it was published.

When new members are appointed to our funding committees, we ensure that the principles of DORA are emphasised to them in their appointment letter, at an annual induction day, and every time they receive papers to review.

How has CRUK changed its application process?

We now ask researchers to list three to five key achievements or contributions to science when applying for our grants. This allows them to describe all types of output relevant to their research, whether it’s datasets, preprints, consensus statements, or anything else. We still collect information about publications but absolutely recognise that this is only one type of research output and encourage researchers to think broadly about the types of outputs that arise from their research.

Open research is a key priority of DORA; how is CRUK promoting open research?

We want to ensure that any data arising from the research that we fund is managed and made available as widely and freely as possible to maximise public benefit. Not only is this good research practice, but we believe it will help us to reach our goal of 3 in 4 patients surviving cancer by 2034. In 2009 we introduced our data sharing and preservation strategy, stating that applicants must provide a data management and sharing plan as part of their application for funding. Data must be shared in a timely and responsible manner and researchers who are providing and receiving data are required to adhere to any relevant regulatory requirements including those relating to the ethical use of data. This plan is reviewed as part of the funding decision, and it is also taken into consideration during milestone reviews and end-of-grant reports.

We also expect our researchers to publish their work on an Open Access basis. We believe this maximises the impact of the research we fund, and facilitates rapid sharing of knowledge ultimately ensuring that patients can access better treatments sooner. We also provide funding for our researchers to publish on an Open Access basis.

How do you see evaluation changing in the future?

Our Research Evaluation team at CRUK have been working hard to transform the way we evaluate the research we fund. Researchfish is providing a us with a wide array of data on the output of our research funding – publications, but also information on further funding, production and re-use of datasets, translational outputs, impacts on policy and many more. We want to understand the impact of our research through a variety of measures, so we are actively engaging with stakeholders to improve our evaluation techniques and investing in new technologies and data sources to help us do this.

What’s CRUK’s position on preprints?

We believe posting preprints on public servers will speed up the dissemination of research findings, give credit to researchers who can claim their work early, and ultimately accelerate and improve scientific progress. We therefore strongly encourage our researchers to deposit preprints of their publications and to cite them, and other non-traditional research outputs in funding applications.

CRUK recently funded a DORA community manager; can you tell us a little bit more about the potential benefits this will bring?

Towards the end of last year, we collaborated with other funders and publishers to hire a DORA Community Manager, who is now in post in Washington D.C. Their sole purpose is to promote DORA, as well as helping signatories with implementation planning. We hope that this post will strengthen ties between funding agencies, publishers and research institutions, recruit new signatories to DORA and build a strong social media presence. We envisage this leading to a greater acknowledgement of DORA principles in the research community, and the establishment of a valuable resource where examples of real progress can be collected and best practice documented.       

Do you think DORA will affect patients in the future?

Patient benefit is always at the core of everything we do at CRUK. We want to fund research that has the potential to positively impact patients, be it through the discovery of new therapeutics, the implementation of innovative treatment regimens, or changes to policy or practice. These research outputs do not necessarily result in publications, so during the evaluation of grants, we now place a large emphasis on assessing how the research that we fund has, or will benefit patients. Further, we now ask all our applicants to highlight the potential of their research proposal to benefit patients in the future.

Find out more

When you apply to us for research funding, your application will be taken through a thorough peer review assessment process.