No.018 - Toward new methods of evaluating humanities research

Toward new methods of evaluating humanities research

Yuki Konagaya

Executive Director, National Institutes for the Humanities (NIHU)

Nowadays, when evaluating the research record of an individual or the research　capacity of an organization, surveyors generally rely on bibliometric methods. However, bibliometric method may not produce accurate evaluations when applied to humanities research. What, then, must we do to ensure appropriate evaluations in this field? Providing just such a solution to this major issue is one task before the National Institutes for the Humanities, or NIHU.

Our organization is tackling this issue in three ways. First, we are striving to improve the bibliometric methods in use. Second, we are developing new, non-bibliometric methods that can be used in evaluations. And third, we are exploring ways to enhance the reputation of NIHU’s humanities research, with the hope of sharing our best practices with the humanities community beyond NIHU.

Humanities research fundamentally takes human society as its subject. For this reason, it must represent a response not just to the concerns of academia, but to society itself as well. The results of the humanities research therefore often appear as books for broader consumption. The focus on producing research output in this format leads to a need for evaluations of the books that are published. When compiling data on the research performance of each of NIHU’s six institutes, (institutional research), we count individual chapters of book-length publications as equivalent to academic papers to ensure equitable comparison with research output in other fields. Researchers serving as coauthors or editors of works also have their contributions counted in this way.

In humanities research—particularly in research on Japan—it is frequently considered important to publish in Japanese in the interest of maintaining a high level of specialized expertise. This leads to the need to evaluate research results published in the Japanese language, and therefore to the need for an environment where such results can be made available online. In this connection, in 2015 we built a repository for all publications produced by NIHU’s six institutes. We also plan to obtain digital object identifiers (DOIs) for all publications, thus clarifying the ways in which they are cited by other researchers.

The humanities are not a research field whose results are concentrated in bleeding-edge subject areas, but rather one displaying considerable dispersion in its interests and focuses. Furthermore, publications in the humanities are likely to be cited for considerably longer than output in other realms of research. The long-lived nature of this research prompted us to investigate the timeframes involved for citations of papers in the NIHU repository; our findings are shown in the figure below. Compared to papers in the information science field, research in the humanities tends to attract its citations over a span around twice as long. In other words, humanities research has a much longer useful life. We are building on these findings by reporting them to inform discussions on appropriate ways to evaluate humanities research, as well as recommending metrics that can take into account the extended citation period.

To evaluate the records of individual researchers, universities generally make use of information on the relative rankings of the journals where their papers are accepted for publication. This approach can, however, downplay the significance of the schools’ own academic bulletins, because these publications will end up being ranked relatively lower compared to academic journals. However, for the humanities in particular—a research realm that rarely pursues the latest trends—academic work that makes steady contributions to the field, such as by adding to the store of available analysis on relevant materials, is also vital. In this connection, universities’ in-house bulletins have a key role to play. This is why we have decided not to rank universities’ in-house bulletins when evaluating NIHU’s research performance.

In our development of non-bibliometric methods for research evaluation, we are attempting to sketch out a complete “cosmos” of research results. The Science Map developed by the National Institute of Informatics, is an effort to chart relationships among scientific research papers based on publication titles. In the humanities, however, it can be much more difficult to elucidate similarities among research efforts based solely on titles in this manner. Our approach is to scan the complete content of the academic papers, mechanically judge their degree of closeness to other publications, and produce a two-dimensional map of the relationships thus clarified. (This work is scheduled for completion in March 2018.) Being able to plot the content of humanities research should allow us to discover the diversity, novelty, and confluence of ideas at play in this cosmos.

Publishing activities play a vital role in building the reputation of the humanities as a field. We are working together with the publishing industry to explore the best forms for academic texts to take in order to respond to the needs of academia while also supporting the publication of general-readership texts that meet the needs of society. During the 2017 academic year, these efforts are leading to the publication of two new books by Heibonsha Ltd.

Moving forward, we aim to invite researchers to come to Japan and experience the research environment here, sharing our output directly with our international counterparts. In this way we also hope to raise the visibility of NIHU’s research among our international colleagues.

Figure: Citation Periods for Research in Various Fields