Responsible Use Of Publication Metrics
Aston University recognises the importance of assessing research outputs to evaluate individual contributions and the performance of organisational units such as Colleges, Departments, and URIs. Publication metrics can provide additional insights into research quality and productivity. However, it is essential to emphasise that the use of such metrics should be guided by ethical considerations, transparency, and the recognition of the primacy of peer review in evaluating scholarly contributions.
In all cases, the University will be guided by its commitment to the responsible use of metrics under DORA, the principles of The Metric Tide report, and its obligations under GDPR.
In addition, the University is mindful of the following limitations and biases:
- Output metrics are not necessarily correlated with originality, significance and rigour in REF
- There is clear evidence that citations and output metrics favour men
- Citations and output metrics can be influenced by career stage, and we have a duty of support for Early Career Researchers
- Citations demonstrably favour anglophone and western-based outlets
- There are different disciplinary norms: thus, monographs are not currently captured by Scopus; moreover, its coverage limited in several disciplines
In recognition of the unique characteristics and data limitations associated with publications in their fields, the School of Social Sciences and Humanities and Aston Law School shall be exempt from the use of publication metrics. This exemption shall be periodically reviewed to assess whether changes in data availability or research practices warrant a revision of the exemption status. Aston University remains committed to promoting research excellence and integrity across all academic disciplines, including the exempted fields.
For more information, please contact: rke_ris@aston.ac.uk
- Why Aston will use metrics:
- To supplement peer review (which will remain the primary recourse for research assessment, as is the case for REF)
- To improve independence, reliability and transparency of research (output) assessment
- To support researchers in monitoring their research performance trajectory against an agreed framework
- How Aston will use metrics: Principles
- To supplement peer review. Publication metrics will be used at the moderation stage of output review only (excluding ALS and SSH).
- To use multiple data points (a basket of metrics), rather than reliance on a single metric.
- To focus on outputs and groups of individuals, but not individuals, as the unit of analysis
- To be transparent about what metrics will be used and for what purpose.
A revised framework for responsible metrics:
Responsible metrics are founded on the principle that the qualities of research reside both in the outputs and impacts of research work, and in the way it is conducted. They have the following dimensions:
- Robustness: basing metrics on the best possible data in terms of accuracy and scope; Humility: recognising that quantitative indicators should not supplant qualitative, expert assessment, but should be used where appropriate to strengthen or complement peer review;
- Transparency: opening up data collection and analytical processes, so those being evaluated are included in the design of the evaluations and can test and verify the results;
- Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research, of research and research-enabling 3 staff characteristics, and researcher career paths across the system;
- Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.
Aston's Baskets of Metrics
- Individual Outputs (metrics over a five-year period)
- Citation Count
- Field-Weighted Citation Impact
- Outputs in Top Citation Percentiles, per percentile
- Patent Count
Where individual outputs are assessed, the following principles will be adhered to:
- Publication metrics shall be considered and utilised solely during the moderation stage of peer review.
- The Policy will be appended with relevant, up to date information and resources about the responsible use of metrics. It will be the responsibility of DDRs to ensure that all those involved in the moderation process are fully appraised of issues pertaining to the responsible use of metrics.
- Metrics are to be regarded as supplementary information, providing an additional layer of context and insight into the quality of research outputs.
- The use of publication metrics at the moderation stage shall be conducted transparently and with accountability. Reviewers must document the role and rationale for using metrics in their assessments.
- Reviewers and evaluators are expected to exercise responsibility when interpreting and using publication metrics. They should be aware of quantitative indicators' potential limitations and biases, as set out above.
- Reviewers shall consider the disciplinary and contextual variations in research practices and outputs when incorporating metrics into their assessments.
- Organisational Unit - College/Department/URI (+1000 outputs)
- Scholarly Output
- Citation Count
- Field-Weighted Citation Impact
- Citations per Publication
- Outputs in Top Citation Percentiles, per percentile
- Publications by Journal Quartile
- Collaboration (International/national/institutional)
- Sector Collaboration (Academic/Industrial or corporate/Third Sector)
- Patent Count
Responsibilities:
- Reviewers, moderators, and line managers are responsible for adhering to this policy and ensuring that metrics are used responsibly. DDRs are responsible for ensuring that reviewers and moderators have access to relevant material.
- Academic units shall provide clear guidance to reviewers regarding the use of metrics in the moderation process.
- The University Research Committee will periodically review and update this policy to align with evolving best practices in research assessment.
- Glossary of Terms
- Scholarly Output
Scholarly Output indicates the prolificacy of an organisational unit: how many publications does this unit have indexed in Scopus?
NB: Care should be taken when comparing organisational units of differing sizes and disciplinary profiles.- Field-Weighted Citation Impact
Field-Weighted Citation Impact (FWCI) indicates how the number of citations received by an entity’s publications compares with the average number of citations received by all other similar publications in the data universe: how do the citations received by this entity’s publications compare with the world average?
Mathematical Notation:
Where:ci = citations received by publication 'i' in the publication year plus the following three years.
ei = expected number of citations received by all similar publications in the publication year plus the following three years.
NB: Be careful using this metric when the entity has a small number of publications. A few highly cited publications can skew the FWCI value.- Citations per publication
Citations per Publication indicate the average citation impact of each of an entity’s publications: how many citations have this entity’s publications received on average?
NB: One or a few highly cited publications can conceal a sizeable body of uncited materials.- Citation Count
Citation Count indicates the total citation impact of an entity: how many citations have this entity’s publications received?
NB: Care should be taken when comparing organisational units of differing sizes and disciplinary profiles.- Collaboration (Geographical / Sectoral)
Geographical Collaboration indicates the extent to which an entity’s publications have international, national, or institutional co-authorship and single authorship. This number is a count unless the percentage symbol (%) is visible.
Academic-Corporate Collaboration in SciVal indicates the degree of collaboration between academic and corporate affiliations: to what extent are this entity’s publications co-authored across the academic and corporate or industrial sectors?
NB: Care should be taken when comparing small organisational units or those of differing disciplines.- Patent Count
This is the count of Patents citing the Scholarly Output published by the entity (e.g. a university) in which you are looking. The count of Patents may be higher than the number of Scholarly Outputs cited, since multiple Patents could refer to the same piece of output. The count of outputs may be higher than the number of Patents since one patent can refer to multiple Scholarly Outputs.
NB: Avoid comparing disciplines that are not likely to have research referenced in a patent- Outputs in Top Citation Percentiles
Outputs in Top Citation Percentiles indicate the extent to which an entity’s publications are present in the most-cited percentiles of a data universe: how many publications are in the top 1%, 5%, 10% or 25% of the most-cited publications?
NB: Care should be taken when comparing organisational units of different disciplines.- Publications in Top Journal Percentiles
Publications in Top Journal Percentiles (SNIP) indicate the extent to which an entity’s publications are present in the most-cited journals in the data universe: how many publications are in the top 1%, 5%, 10% or 25% of the most-cited journals indexed by Scopus?
NB: Care should be taken when the objective is to judge an organisational unit's publications based on their actual performance rather than the journal average.
It is important to be aware that a role as a journal editor will skew these numbers.