FAQ (Frequently Asked Questions) on Performance Ranking

The features of this performance ranking are that 80% are qualitative indicators (the first time using h-index to reflect on the quality of universities’ papers), and that long-term and short-tem indicators each weights up to 50%, thereby emphasizing the outcome of short-term efforts. However, this performance ranking focused on the qualitative and quantitative performance of scientific papers, and did not include other frequently used university evaluation indices such as teaching, research, and administration, nor did it emphasize on academic performance indices such as reputation and extraordinary achievements. The indices designed for this performance ranking study were suitable as reference especially for research-oriented universities in newly developed countries.

This performance ranking targeted research-oriented universities, especially those in newly developed countries. Through objective indicators which would also reflect short-term efforts, each university would be able to understand its position and advantages in the world rankings, and from there it would know how it fares against other universities, and it can track its annual progress in terms of the quality and quantity of its scientific papers.

No. This performance ranking targeted only scientific papers from universities around the world. The indicators were designed to compare the quality and quantity, long-term and short-term respectively, of universities’ scientific and social science papers. These indicators could not be applied to overall evaluative indices including teaching, research, and administration, etc.

Not completely. In addition to scientific papers, there are other aspects comprised in the overall academic capabilities of a university, such as research projects, patents, monographs, and collaborations with industry, etc. However, with the exception of humanities disciplines, the quality and quantity of scientific papers are well-received indicators for assessing the academic capability of a university.

The “Academic Ranking of World Universities” published by Shanghai Jiaotong University uses indicators which include the number of Nobel laureates affiliated with that institution, the number of highly cited scholars, and numbers of papers in Nature or Science. They are after extraordinary research output. However, these and especially the Nobel Prize are goals beyond most universities could achieve, and therefore could not effectively reflect the research performance of most universities. HEEACT’s performance ranking project takes into account a set of indicators which was designed to include short-term indicators which evaluate universities’ present efforts. As long as there is progress, it would be reflected on the ranking of that university in the following year.

The “World University Rankings” by the Times Higher Education Supplement emphasizes on peer review and employer review, weighing up to 50%. The results could easily be influenced by subjective views, especially in its questionnaire-type ranking and point allotment, which would easily turn this into an assessment of a university’s reputation and not its actual performance. HEEACT’s performance ranking project utilizes more objective methods and statistics to conduct its university ranking.

Yes. The number of papers from a university is naturally related to the size and history of that university. Therefore this performance ranking project uses the average citations per paper to balance the influence of a university’s size and history. To a certain extent, the size and history of a university do represent its strength and ability, so in this performance ranking project, we have tried to find a balance point. In next year’s performance ranking project, we will continue to lessen the impact of size and history.

In liberal arts disciplines, the research tends to take on regional characteristics, and research output is often published in non-English journals or via monographs. Since the A&HCI mainly indexes papers in English journals, therefore this performance ranking project excludes journals papers from A&HCI..

The h-index is a highly sensitive indicator, and the number of papers for two years is already quite high (for example Harvard University has 28,951 papers in two years), and is enough to create an objective qualitative and quantitative index. We have taken universities with the highest numbers of papers for eleven years—Harvard University and Tokyo University—along with universities dispersed around the mid- and lower numbers—all together 47 universities in all. We applied long-term and short-tem h-index correlation coefficient analysis, and discovered that the correlation coefficient of 2-year h-index and 11-year h-index were 0.967, and were highly correlated. Therefore there was no need to conduct 11-year h-index calculation.

Since this performance ranking is focused on the qualitative and quantitative study of scientific papers, it could only reflect on the qualitative and quantitative performance of these papers from each university at the time. This is not an “academic ranking” or a “university ranking.” Therefore, the results might vary somewhat with other university rankings. Different indicators, the size and history of a university, whether there are medical schools, and the percentage of liberal arts and social science programs in a university are all factors that might affect the ranking outcome.

Yes. Since the first-place university in each indicator has significantly higher score than other universities, the differences in total scores for many universities with lower rankings are rather close. The ranking given in this project is aimed at giving the reader an approximate order and does not intent to show any absolute disparities between universities. Therefore, it is not suitable to single out those universities which are close in scores and rankings by giving them an absolute rank to make conclusive judgment on their performance. Instead, they should be viewed on par with other universities that demonstrated similar performance in their scientific papers.

No. Lower ranking means that a university’s scientific papers did not do well in terms of qualitative and quantitative performance when compared with other universities. This could not and should not be over-generalized to its overall academic performance or even to its academic reputation. When dealing with any ranking, it is important to note the differences in indicators used, the size and history of the universities, whether there are medical schools involved, or even whether humanities and social science disciplines make up the majority of certain universities, etc.

We would encounter problems with setting the range of each group, if we had indicated those universities closer in scores simply into two ranking groups. What might ensue is that there might be only a slight difference in scores between the university at the last place of the first group and the university at the first place of the second group, yet these two universities are ranked into different groups. Since it would be difficult to determine how many universities should be included in a ranking group, and slight score differences should not be ranked into two groups, thereby HEEACT’s performance ranking project did not indicate additional ranking by groups.

This performance ranking is based on data that were obtained objectively to study and evaluate the qualitative and quantitative performance of universities’ scientific papers. In general when conferring upon the performance of scientific papers, quantitative analytical indicators would better provide the basis for more objective characteristics, especially since quantitative indicators are more suitable for ranking purposes. Although this performance ranking mainly used quantitative indicators, we used it in conjunction with concepts that would observe the quality of the papers, so in all 80% of the indicators were quality based. Ours could be termed a quantitative ranking that is based on qualitative assessment.

Yes. This performance ranking project studied scientific journal papers indexed in SCI and SSCI databases, and did not include humanities journals (A&HCI database). At the same time, even though we included papers in social sciences, but since the number of social science journals is relatively fewer (SCI indexes 6,771 journals while SSCI indexes 2,005 journals), if a university is mainly known for its humanities and social science programs and does not have a relatively stronger science program, then it would be difficult for that university to place in this ranking.

Yes. Papers published from the medical sciences and their citations are significantly higher than many other disciplines. From 1997 to August 2007, there were 1,868,486 published papers in clinical medicine, and these were cited 20,643,979 times; while there were 723,371 published papers in engineering, and those were cited 2,542,146 times. Therefore in this ranking, the performances of universities with medical schools or hospitals score better than those universities without medical related programs. In the future we will consider conducting additional annual rankings based on different academic disciplines.

Yes. Even though this performance ranking used indicators such as the average citations per paper to balance the impact of a university’s size, but in the final ranking, it is still evident that the size of the university made its impact. For example, the University of Washington in Seattle has a student body of approximately 40,000 (and a faculty body of 5,700) and a medical school, so its rank was quite high. This year we have attempted to lessen the impact of university size, but apparently still not enough. Next year we will continue to lessen the impact of university size on the performance ranking.

This performance ranking is not a “reputation ranking” nor an “academic ranking” of universities, so some people might feel that the outcome might be different from what they had come to know and expect. In addition, for those universities in a given country that were included in the ranking, there might have been some discrepancies in their actual rank and what was expected of them, but the relative position in rank between universities of the same country may still match society’s expectations.

The correlation coefficient between the total scores of HEEACT’s and the Shanghai Jiaotong University was 0.896, and the correlation coefficient of the ranks was 0.853. In addition, nine of our indicators have statistically significant correlation with the total scores from Shanghai Jiaotong University. In regression model, they are all significant variables. In other words, consideration of short-term indicators have resulted in the increase in the number of ranking indicators, but this would allow a university’s efforts to be visible in the next year.

That there is a significant difference in scores between the first-place and second-place is a prevalent phenomenon in academic ranking. The difference in scores between first-place and second-place in the Shanghai Jiaotong University ranking was approximately 30 points. The Times, in wanting to lessen the gap in scores, used the Z-scores as adjustment; that decrease in score gap was the result of manmade adjustment. In conducting this performance ranking, we had attempted to use the T-scores to adjust and lessen the score difference between first-place and second-place. However in the end, we wanted to adhere to and present the original point difference, so we decided not to do any points adjustments.

The first-placed Harvard University is large in size (student body of 25,000 and faculty of 2,500) and has a medical school with great performance record (from 1996-2006 there were 35,492 published papers on clinical medicine and these were cited 904,586 times), and there was also exceptional qualitative and quantitative performance of papers from its other academic disciplines. Out of the nine indicators, with the exception of two (average number of citations and fields of excellence), Harvard received the highest points, and its performance fared outstanding when compared with other universities. The second-placed Johns Hopkins University is also a large-sized university (student body of 20,000 and faculty of 3,100) and has a medical school with numerous medical science papers (from 1996-2006 there were 21,205 published papers on clinical medicine and these were cited 487,568 times), yet its long-term and short-term number of papers and citations were significantly behind Harvard. In addition, its number of highly cited papers and its number of high-impact journal papers were both significantly behind Harvard, so this is the reason that there was a significant gap in total score between the first-place and second-place universities.