This article discusses the author’s experience of being angered by a “divine paper” that was released yesterday.
This “divine paper” is “How not to Lie with a Benchmark: Rearranging NLP Leaderboards”. The general content of the paper suggests that many current leaderboards use the arithmetic mean for averaging, whereas it argues that the geometric mean and harmonic mean are more reasonable. Most importantly, it recalculated the rankings of models on leaderboards like GLUE and SuperGLUE using geometric and harmonic means. The results supposedly showed that models which previously surpassed human performance no longer did so under the new averaging schemes.
Does it sound interesting? I thought it was quite interesting too, so I planned to write a blog post to introduce it. However, as I was finishing the blog post and cross-checking the data, I discovered that the data in the tables was completely fabricated!!! The actual results do not support its conclusions at all!!! Therefore, this blog post has turned from a “commendation meeting” into a “criticism meeting”...
Pure Nonsense
First, let’s present the first table from this “divine paper,” which concerns some results from the GLUE leaderboard:
Setting everything else aside, the fact that this “divine paper” confuses “,” (half-width comma) and “.” (decimal point) in its tables is disgusting enough (the SuperGLUE table below is even worse). But if it were just such minor issues, I could tolerate them. What is most intolerable is: the calculation rules for AM (Arithmetic Mean), GM (Geometric Mean), and HM (Harmonic Mean) in it are simply “arbitrary”!
I tried for a long time and finally figured out the calculation rules for this table:
1. All AMs are calculated using the first 10 tasks (even though the table above only shows the results for the first 8 tasks);
2. The GM and HM for the “Human” row are calculated using the first 10 tasks;
3. The GM and HM for the models in other rows are calculated using all 11 tasks.
Since the performance on the 11th task is lower than the others, the GM and HM calculated for the models this way are lower than those for Humans. The author then directly concluded that under GM and HM, human performance is still number one. In fact, if everyone were evaluated using the same set of tasks, there would be basically no difference in the rankings for AM, GM, and HM. Moreover, anyone with a slightly normal mathematical mind can see the impropriety of the above results: on many tasks, model performance far exceeds Humans, and on a few tasks, models are inferior to Humans but only by a little bit. Therefore, as long as it is a normal averaging algorithm, it is impossible to conclude that Humans far exceed the models, right? Yet, the author actually believed it...
The same error also appears in SuperGLUE:
Its calculation rules are:
1. All AMs are calculated using the first 8 tasks;
2. All GMs and HMs are calculated using all 10 tasks.
In fact, if the AM were also calculated using the results of 10 tasks, then according to the AM ranking, Humans would also be number one. That is to say, as long as everyone’s calculation standards are the same, there is no significant difference in the rankings of AM, GM, and HM.
Truly Helpless
By the way, this paper was also accepted into a NeurIPS 2021 Workshop. Although Workshops are usually far inferior to formal papers, they shouldn’t be a mess to this extent. Looking at the title of this paper again, I wonder if it would be more appropriate to change it to “How not to Lie with this paper”?
It seems that in the future, when we read papers, we should not only care about the reproducibility of the results but also pay attention to whether their sums, means, variances, etc., are calculated correctly\sim Truly, all kinds of bizarre possibilities exist\sim
When reposting, please include the original address of this article: https://kexue.fm/archives/8783
For more detailed reposting matters, please refer to: “Scientific Space FAQ”