In the article “You May Not Need BERT-flow: A Linear Transformation Comparable to BERT-flow”, inspired by BERT-flow, I proposed an alternative solution called BERT-whitening. It is simpler than BERT-flow but achieves similar or even better results on most datasets. Additionally, it can be used for dimensionality reduction of sentence vectors to improve retrieval speed. Later, together with several collaborators, I supplemented the experiments for BERT-whitening and wrote it into an English paper titled “Whitening Sentence Representations for Better Semantics and Faster Retrieval”, which was released on arXiv on March 29 of this year.
However, about a week later, a paper titled “WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach” (hereinafter referred to as WhiteningBERT) appeared on arXiv. Its content highly overlaps with BERT-whitening. Some readers, upon seeing this, provided feedback to me suggesting that WhiteningBERT might have plagiarized BERT-whitening. This article serves as a report to concerned readers regarding the results of my communication with the authors of WhiteningBERT.
Timeline
First, let’s review the relevant timeline of BERT-whitening to help everyone straighten out the sequence of events:
January 11, 2021: Published the article “You May Not Need BERT-flow: A Linear Transformation Comparable to BERT-flow” on this blog, proposing BERT-whitening for the first time. At this time, the article did not yet include the dimensionality reduction part.
January 19, 2021: The BERT-whitening blog post was forwarded to the WeChat official account “Xi Xiao Yao’s Cute House” (Link). After being published on both the blog and the official account, I believed that BERT-whitening had spread quite widely, at least within the domestic NLP circle.
January 20, 2021: Student Liu, a researcher at Tencent, pointed out to me that BERT-whitening is essentially a PCA, so it could also be used for dimensionality reduction. After testing, the sentence vectors after dimensionality reduction even showed improvements in some tasks—being both fast and effective. Therefore, I updated this content in the blog.
January 23, 2021: Feeling that BERT-whitening had some academic value, I invited Student Liu and Student Cao to supplement experiments and planned to write an English paper to submit to ACL 2021. At that time, there was only about a week left before the deadline.
February 02, 2021: Fortunately, we finished both the experiments and the paper and submitted it before the ACL 2021 deadline.
March 26, 2021: The review results for ACL 2021 came out. We felt they were not very optimistic and didn’t bother with a rebuttal, so we planned to put the paper directly on arXiv.
March 29, 2021: The English paper for BERT-whitening, “Whitening Sentence Representations for Better Semantics and Faster Retrieval”, was released on arXiv.
April 05, 2021: The WhiteningBERT paper “WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach” appeared on arXiv.
September 26, 2021: The Accepted Papers for EMNLP 2021 were announced, confirming that WhiteningBERT was accepted to EMNLP 2021.
Readers might wonder why, after half a year since April 5, I am only bringing this up now. First, because the BERT-whitening method is relatively simple, I did not rule out the possibility that others independently achieved the same results. Therefore, I didn’t pay much attention when WhiteningBERT first appeared on arXiv. Secondly, even in the worst-case scenario—assuming (merely assuming) that WhiteningBERT did copy BERT-whitening—it was just a minor issue on arXiv and not a big deal, so there was no need to waste time on it.
However, once I learned that WhiteningBERT was accepted to EMNLP 2021, the nature of the matter was no longer a “minor issue.” I decided to try communicating with the authors of WhiteningBERT, hoping they could prove the originality of WhiteningBERT to avoid unnecessary misunderstandings. Below is our communication process.
Email Communication
On September 26, I sent the first email to all authors of WhiteningBERT:
Hello authors,
First, congratulations on your work “WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach” being accepted to EMNLP 21.
However, I have found that your work is almost identical in methodology to the blog post I published on January 11, 2021 (https://kexue.fm/archives/8069), and even the final naming of the method is almost identical. Therefore, I have reason to doubt the originality of your work’s methodology.
Thus, I believe it is necessary for you to provide evidence showing that your work is indeed independent and original (such as manuscript editing records to prove your work started before January 11). If not, I request that you withdraw the paper from EMNLP and issue a public apology. If there is no response to these two points, I will have no choice but to initiate a public discussion online.
Looking forward to your reply.
Because I was quite emotional after just learning the news, the wording was not very friendly; please excuse me for that. Shortly after that day, the first author of WhiteningBERT replied to me:
Hello,
We have received your letter. Thank you for your interest in our work!
First, we believe that our work (submitted to arXiv on April 5, 2021) and your “Whitening Sentence Representations for Better Semantics and Faster Retrieval” (submitted to arXiv on March 29, 2021) are concurrent works. The two papers have similarities, but the points we want to declare and the stories we tell are not almost identical. This is mentioned and cited in our paper.
Second, regarding research on unsupervised sentence representations, we were already conducting this as early as last year. We hoped to obtain sentence representations unsupervisedly based on existing pre-trained models and explored various methods such as inter-layer combinations, data augmentation, introducing graph structures, linear transformations, pre-training, knowledge transfer, etc. We experimented on several sentence semantic similarity tasks. Some methods that did not work were not mentioned, and finally, we reached three simple and useful conclusions, summarized into this experimental paper. As for the final name WhiteningBERT, it was chosen because one of our methods used PCA Whitening. While the name might seem like clickbait, it was renamed WhiteningBERT when writing the paper for ease of reference. (Our initial name was MatchingBERT; the figure below shows that the last modification time for some files therein was in July 2020.)
Third, regarding the originality of the method, we did not claim that the PCA Whitening algorithm was our original invention. In fact, the three methods in our conclusion are all very simple, and many papers and tutorials have introduced whitening methods. Therefore, we admit that the novelty is limited.
Finally, regarding the similarity in content you mentioned (including the blog you mentioned) and the lack of innovation, these issues were already raised and discussed by reviewers during the EMNLP 2021 review process. The PC and finally the SPC were aware of the whole situation. But in the end, they still decided to accept it. I think our work still has value that the PCs recognized.
Best regards.
The reply also included two screenshots:
At this point, I was quite grateful that the first author was willing to communicate actively on this issue. However, the first author’s reply did not eliminate my doubts, so I replied on the same day:
Hello,
Thank you for your reply. However, what I am questioning is not your innovation, but your originality.
1. I know many people at Microsoft specifically research various NLP tasks, but this does not negate my doubts.
2. Screenshot 1 can only serve as very weak proof that you were working on something called “MatchingBERT” early on, but I cannot determine the actual content of MatchingBERT itself.
3. Screenshot 2 also cannot negate my doubts.As for “the PC and finally the SPC were aware of the whole situation,” do you mean that the PC and SPC, while knowing the premise that “a Chinese blog post introduced the same method more than two months before WhiteningBERT was submitted to arXiv, and an English paper introduced the same method a week before WhiteningBERT was submitted to arXiv,” still did not question your originality and accepted it?
The above exchanges all took place within September 26. After that, until October 5, I still had not received any response from any of the authors of WhiteningBERT. So I emailed all the authors again to inquire:
Hello authors, sorry to disturb your National Day holiday.
After I raised my doubts, the first author replied to my email on the same day, and I immediately replied to the first author’s email. The content of the exchange is attached below. But after my reply, up to now, I have not received any subsequent response from any author. In the spirit of science, I do not want to cause any misunderstanding, so I hope to further confirm the relevant situation of this matter. So I am taking the liberty to disturb you again to ask if you have decided not to make any further response to this matter?
Soon, the first author replied to my email:
Hello,
We have currently requested the company’s legal department to conduct an assessment, and the legal department will provide a response. Since it is currently the National Day holiday, we hope you will understand!
Best regards.
Personal View
To be honest, receiving this email from the first author left me with very complex feelings—shock, confusion, and a bit of speechlessness. Originally, I wasn’t sure about the nature of the matter, so I decided to email and inquire first to avoid later misunderstandings and embarrassment. If the authors could demonstrate the independence of WhiteningBERT’s proposal, it would be a happy ending for everyone, providing an explanation to both the readers and me. Instead, the authors delayed responding directly to the question and turned to consult the legal department. What kind of move is this?
As mentioned in the timeline, when we decided to organize BERT-whitening into a paper and submit it to ACL 2021, there were less than two weeks left before the deadline. Yet, in less than two weeks, we finished the experiments and wrote the paper (though the English level was relatively poor). Therefore, if WhiteningBERT was truly proposed earlier than BERT-whitening, with such a strong lineup of authors, they should have been able to complete the experiments and paper long ago. At the very least, after the BERT-whitening blog was published, they should have released their paper on arXiv to demonstrate their originality. How could it be such a coincidence that they waited until after the BERT-whitening English paper was on arXiv to release their own?
Of course, even with these doubts, we still cannot definitively characterize this matter. The reason is simple: BERT-whitening is so straightforward that the possibility of independently repeating the same work cannot be ruled out. That is why the subsequent email communication took place. So, we are back to the authors’ “mysterious move”—what was the consideration behind transferring this to the legal department?
In fact, it is inherently difficult to have substantive evidence to prove plagiarism in this case. Therefore, even if the authors of WhiteningBERT made no response at all, there would be no legal risk. The reason for hoping the authors could provide some proof was purely a moral appeal, not an attempt to “bring anyone to justice.” This is a scientific issue, not a PR issue. Therefore, even if the legal department can eliminate the authors’ legal risks, if the authors remain unwilling to provide substantive proof, how can the doubts in the hearts of the readers and myself be eliminated?
Please include the original address of this article when reprinting: https://kexue.fm/archives/8715
For more detailed reprinting matters, please refer to: “Scientific Space FAQ”