How Indonesia’s ranking system is driving researcher performance

For a country that is the fourth largest populous country in the world, Indonesia has historically struggle to compete in the global knowledge production. 

A study by Caroline Fry, John Lynham and Shannon Tran from the University of Hawaii has shed light on Indonesia’s remarkable transformation in recent years. 

In just three years, Indonesia went from being the second worst to the top producer of scientific journal articles in Southeast Asia. 

What contributed to this significant turnaround in Indonesia’s research output? 

The study investigated whether a transparent system of ranking every single researcher in the country based on publications and citations, called SINTA, played a role in this remarkable transformation.

SINTA is an online database that indexes research publications and assigns individual “research scores” to nearly every researcher in Indonesia. 

It was designed to allocate higher scoring weights for publications in high-impact Scopus-indexed journals. 

The scoring formula was loosely modelled on the European Commission’s ACUMEN project, which aimed to understand the ways in which researchers are evaluated by their peers and institutions.

Using panel data from over 200,000 Indonesian researchers and comparing them to researchers from Thailand and the Philippines, the study found that the implementation of SINTA coincided with changes in the production of publications by Indonesian researchers consistent with the weights used in the ranking formula. 

The study showed that SINTA played a role in increasing publication counts in Indonesia and contributed to overall improvements in scientific capacity in low- and middle-income countries.

After the implementation of SINTA, Indonesia exhibited a sharp increase in researcher publication output, and Indonesian institutions rose in global rankings that rely on Scopus-indexed publications—a trend unrivalled by neighbouring Southeast Asian countries. 

Indonesian researchers increased their publication rates in high-impact international journals and produced significantly more “non-journal” articles, such as conference proceedings, relative to peer-reviewed articles.

The study also found that Indonesian researchers expanded their collaboration networks, both foreign and local, which is an important element of improving scientific capacity in low- and middle-income countries.

However, the study also showed some evidence of tendencies to “game the system” under SINTA’s current scoring framework. 

Nonetheless, the study demonstrates that researchers in Indonesia may respond strongly to non-financial incentives.

The evidence so far points to SINTA contributing to an increase in total Scopus publications in Indonesia, albeit with some evidence of quantity for quality substitutions. 

The potential policy implications of a nationwide scientific ranking system are complex. 

A primary benefit of implementing a ranking system is its potential to influence research efforts and priorities at a low cost.

With the migration of talented researchers into higher-paying countries and a widening global knowledge gap, non-costly ranking systems could be especially valuable for low- and middle-income countries that hope to boost research interest and retention among local researchers.

The he study provides empirical evidence that SINTA contributed towards an increase in publication output in Indonesia and shows that ranking and evaluation systems for researchers can contribute to overall improvements in scientific capacity in low- and middle-income countries.

Read the research article: Caroline Fry , John Lynham , Shannon Tran, 2023, Ranking researchers: Evidence from Indonesia, Research Policy, Vol. 52, Issue 5.