NRLgate -
Plagiarism by Peer Reviewers

Home Page

Abbreviated Table of Contents

1. Background Information

2. Evidence of plagiarism in reviews A and B of a paper on empirical discovery submitted to the Machine Learning Conference

3. Evidence of plagiarism in reviews X and Y of a paper on optimal control strategies submitted to the Machine Learning Conference

4. Evidence of plagiarism in reviews #1, #2, and #3 of a paper on electrical circuit design submitted to the Evolutionary Computation journal

5. Evidence of plagiarism in reviews T2 and T3 of a paper on pursuer-evader games submitted to the Tools for Artificial Intelligence conference (TAI); evidence that TAI reviewers T1, T2, and T3 are affiliated with the Naval Research Laboratory; and similarities linking reviews T1, T2, and T3 with various MLC and ECJ reviews

6. There are only 2 people in the overlap between the reviewers for the Machine Learning Conference (MLC), the editors and editorial board of the Evolutionary Computation journal (ECJ), and Tools for Artificial Intelligence conference (TAI)

7. Indications that there are only 3 (as opposed to 9) different plagiarizing reviewers among the peer reviewers at the Machine Learning Conference (MLC), the editors and members of editorial board of the Evolutionary Computation journal (ECJ), and the Tools for Artificial Intelligence conference (TAI)

8. Preliminary Conclusions

9. Proposal for binding arbitration, under auspices of the American Arbitration Association, by a retired federal judge of the issues and plagiarism and scientific misconduct

10. Public policy and scientific policy issues involving the scientific peer review process and the Naval Research Laboratory

11. References

Linkage Diagram for the Plagiarizing Reviewers

Detailed Table of Contents

1. Background Information

1.1. Genetic Programming (GP)
1.2. The Evolutionary Computation Journal (ECJ)
1.3. The Machine Learning Conference (MLC)
1.4. Tools for Artificial Intelligence Conference (TAI)
1.5. Other conferences in evolutionary computation
1.6. The Naval Research Laboratory (NRL)
1.7. The SAMUEL system developed in-house at the Naval Research Laboratory (NRL)
1.8. The connection between the scientific peer review process and the problem of measuring the internal performance at the Naval Research Laboratory (NRL)

2. Evidence of plagiarism in reviews A and B of a paper on empirical discovery submitted to the Machine Learning Conference

2.1. Both reviewers A and B put quotation marks around the same insignificant and unmemorable 3-word prepositional phrase "In one run, ..."
2.2. Both reviewers A and B made the same punctuation error in adding ellipsis in their quotation of "In one run, ..."
2.3. Both reviewers A and B converged on the same section of the paper review form to locate their quotation of "In one run, ..."
2.4. Both reviewers A and B devoted significant fractions of their reviews to a complaint about a 3-word reference to an operational detail (the Ethernet)
2.5. Both reviewers A and B couched their complaint about the offending 3-word reference to the Ethernet in a 3,118-word paper as a matter of "spending space"
2.6. Both reviewers A and B converged on the same verb in complaining about the matter of "spending space" on the 3-word reference to the Ethernet
2.7. Both reviewers A and B converged on the same section of the paper review form to place their complaint about "spending space" on the 3-word reference to the Ethernet
2.8. Both reviewers A and B both neglected section 5 of the paper review form
2.9. Both reviewers A and B employed the same word while neglecting section 5 of the paper review form
2.10. The opening sentences of both reviews A and B gratuitously provided the same unrequested information
2.11. Both reviewers X and Y substituted "Genetic Algorithms" or "GA" in lieu of the author's chosen term
2.12. Lock-step treatment by both reviewers A and B in section 2 of the paper review form
2.13. Both reviewers A and B used a similar hostile and antagonistic tone
2.14. Both reviewers A and B offered similar advice about statistics, using the same trio of phrases

2.14.1 The joint advice of reviewers A and B was at variance with the prevailing practice concerning empirical discovery

2.15. Reviewers A and B both ignored the intellectual content of the submitted paper
2.16. Reviewer B's quotation of "on one run ..." is inaccurate --- thereby suggesting the time sequence of the plagiarism

3. Evidence of plagiarism in reviews X and Y of a paper on optimal control strategies submitted to the Machine Learning Conference

3.1. Both reviews X and Y contained a 3-part explanatory sentence containing a trio of almost identical phrases
3.2. Both reviewers X and Y complained that the submitted paper does not properly identify the 3 dimensions of the problem
3.3. Both reviewers X and Y put quotation marks around the same words when no apparent reason exists for quoting the words
3.4. Both reviewers X and Y couched their joint complaint about the 3 dimensions in the same semantically unusual way
3.5. The unlikely coincidence that reviewers X and Y would both make the same demonstrable error claiming the absence of a definition that is demonstrably in the submitted paper
3.6. The unlikely coincidence that reviewers X and Y would both make the same mistaken claim concerning the existence of a "standard" version of a particular problem
3.7. The unlikely coincidence that both reviewers X and Y employed the same quaint and infrequently used word in their reviews
3.8. The unlikely coincidence that both reviewers X and Y would abbreviate the paper's title and that they would both abbreviate it to the same 6 words
3.9. The opening sentences of both reviews X and Y gratuitously provided the same unrequested information
3.10. Both reviewers X and Y substituted "genetic algorithm" in lieu of the author's chosen term
3.11. The opening sentences of both reviews X and Y were similar in structure
3.12. The time sequence of the plagiarism is suggested because reviewer Y correctly identified the 3 dimensions of the problem a mere 2 sentences before he complained that the paper failed to properly identify them

4. Evidence of plagiarism in reviews #1, #2, and #3 of a paper on electrical circuit design submitted to the Evolutionary Computation journal

4.1. Reviewer #2 referred, in his review, to a specific comment raised by reviewer #1, in his review --- thereby indicating that reviewer #2 had review #1 in front of him when he wrote his review

4.1.1. Reviewers #1 and #2 both use the word "hard" in a colloquial way in questioning the difficulty of the problem in the submitted paper
4.1.2. Both reviewers A and B converged on the same inappropriate section of the paper review form to locate their question about "hardness"
4.1.3. Reviewer #2 referred, in his review, to a specific comment raised by reviewer #1, in his review
4.1.4. Both reviewer #1 and #2 immediately followed the word "hard" with a clarification of exactly what they meant
4.1.5. Both reviewer #1 and #2 switched to the interrogatory mode at the same point

4.2. The telltale > e-mail symbols indicate that review #1 was transmitted to peer reviewer #2 by e-mail and that reviewer #2 had review #1 in front of him when he wrote his review. This fact is confirmed by the lock-step mirroring of thoughts, words, and phrases by reviewers #1 and #2 in ALL 5 SECTIONS of the journal's paper review form

4.2.1. For section 2, the telltale electronic >'s (as well as groups of mirrored words) indicate that review #1 was supplied by e-mail to peer reviewer #2
4.2.2. For section 3, the telltale electronic >'s (as well as the reference by reviewer #2, in his review, to an issue raised by reviewer #1, in his review) indicate that review #1 was supplied by e-mail to peer reviewer #2
4.2.3. For section 4, the telltale electronic >'s (and similar thoughts) indicate that review #1 was supplied by e-mail to peer reviewer #2
4.2.4. For section 5, the telltale electronic >'s (and similar words in similar locations) indicate that review #1 was supplied by e-mail to peer reviewer #2
4.2.5. For section 1, the telltale electronic >'s indicate that review #1 was supplied by e-mail to peer reviewer #2
4.2.6. It would be difficult for 2 or more members of the geographically dispersed editorial Board of the Evolutionary Computation journal to collude on the peer reviewing of a particular paper without involvement by the editor. None of the ordinary methods of e-mail transmission by reviewer #1 to the editor would cause the >'s to become attached to the responses written by reviewer #1.

4.3. The unlikely coincidence that reviewers #1 and #2 would both make the mistaken assertion that the paper lacked a specific item even though the paper actually devotes almost 2 pages to the item
4.4. The unlikely coincidence that both reviewers #1 and #3 would complain about "advertising"
4.5. The unlikely coincidence that both reviewers #1 and #2 would abbreviate the paper's title and that they would both abbreviate it to the same 8 words
4.6. The unlikely coincidence that both reviewers #1 and #2 would be agitated about the paper's evading what they claimed to be an important issue concerning "constraints" and "bounds"
4.7. The unlikely coincidence that both reviewers #2 and #3 would refer to "Koza," by name, in their reviews
4.8. The unlikely coincidence that reviewers #1, #2, and #3 all shared a hostile and antagonistic tone
4.9. The non-automatically-created time-and-date lines points to an unusual sequence of events in creating these 3 review documents

4.9.1 The absence of the time-of-day indicates that the 3 "date" lines that the submitting authors received from the North American Associate Editor were not the familiar "date plus time-of-day" lines that are automatically created when an e-mail message is transmitted on the Internet.
4.9.2 The unusual format of the 3 "date" lines were not the familiar "date plus time-of-day" lines that are automatically created when an e-mail message is transmitted on the Internet

4.10. The 4 appearances of the submitted paper's title points to an unusual sequence of events in creating the review documents. The 2-line title line of review #3 is an exact "cut-and-paste" copy of the two-line title in the section written by the North American Associate Editor, but the abbreviated 1-line title lines of reviews #1 and #2 were not.

5. Evidence of plagiarism in reviews T2 and T3 of a paper on pursuer-evader games submitted to the Tools for Artificial Intelligence conference (TAI); evidence that TAI reviewers T1, T2, and T3 are affiliated with the Naval Research Laboratory; and similarities linking reviews T1, T2, and T3 with various MLC and ECJ reviews

5.1. Reviewer T1 used quotation marks and ellipsis and the 3-word prepositional phrase "On one run, ..."--- thereby linking TAI reviewer T1 to either MLC reviewer A or B
5.2. Two reviews of different papers at different conferences contain the same quotation error ("on" instead of "in") --- thereby establishing that TAI reviewer T1 is the same person as MLC reviewer B
5.3. Review T2 contained a paragraph that is almost identical in thoughts, words, and phrases to paragraphs contained in reviews A, X, and #2 --- thereby suggesting that TAI reviewer T2 may be the same person as reviewers A, X, and #2
5.4. Review T2 substituted "GA" in lieu of the author's chosen term ("Genetic Programming") in the same way as MLC reviewers A, B, X, and Y
5.5. The opening sentences of TAI review T2 and MLC review A contain 5 similarities
5.6. There are 4 similarities in another sentence of TAI review T2 and MLC review A and they both contain a particular infrequently used word that MLC reviewer X frequently uses
5.7. Reviewers T2, A, and B over-diligently filled in every blank on the paper review form
5.8. Reviewer T3 contained 2 sentences that are almost identical in thoughts, words, and phrases to paragraphs contained in MLC review X and ECJ review #2
5.9. Both reviewers T2 and T3 zeroed-in on the obscure issue as to why the submitted paper did not use the mutation operation
5.10. Both reviewers T2 and T3 were argumentative about the ability of genetic programming for solving problems without the mutation operation
5.11. Reviewers T2, T3, and #3 are the only reviewers who raised the issue of the mutation operation --- thereby suggesting that Evolutionary Computation reviewer #3 is either TAI reviewer T2 or T2

6. There are only 2 people in the overlap between the reviewers for the Machine Learning Conference (MLC), the editors and editorial board of the Evolutionary Computation journal (ECJ), and Tools for Artificial Intelligence conference (TAI)

6.1. There are only 2 people in the overlap between the Machine Learning Conference and the Evolutionary Computation journal
6.2. There are only 2 of the 24 people who were reviewers for the Machine Learning Conference who are involved with the field of evolutionary computation

6.2.1. Reviewers A and X of my MLC papers were knowledgeable about evolutionary computation
6.2.2. Reviewer X of my MLC paper is conversant with the comparative number of lines of computer code in various computer implementations of genetic algorithm

6.3. There are only 2 people in the overlap between the reviewers for the MLC and TAI conferences and there are only 2 people in the overlap between the reviewers for the TAI conference and the editors and editorial board of the Evolutionary Computation journal

7. Indications that there are only 3 (as opposed to 9) different plagiarizing reviewers among the peer reviewers at the Machine Learning Conference (MLC), the editors and members of editorial board of the Evolutionary Computation journal (ECJ), and the Tools for Artificial Intelligence conference (TAI)

7.1. Reviews A, X, #2, the signed non-anonymous review to the MIT Press of my first book, and the North American Associate Editor's transmittal letter employ the word "limitation" with a frequency of 1-in-530; the 316 GP-96 peer reviews use it with a frequency of 1-in-12,822; and the word doesn't appear at all in reviews B, Y, #1, #3, T1, T2, and T3
7.2. Reviewers B, Y, and #1 consistently misspell "LISP"
7.3. Reviews B, Y, #1, and T1 do not contain any typographical or spelling errors (except for "Lisp"), whereas reviews A, X, #2, #3, and the signed, non-anonymous review to the MIT Press of my first book all contain numerous such errors
7.4. Reviewers A, X, and T2 use the particular infrequently used word "judge"
7.5. Review X and the signed non-anonymous review to the MIT Press of my first book used the unusual self-referential style "This reader"
7.6. Reviews A, B, #2, and the signed non-anonymous review to the MIT Press of my first book all used the quaint and infrequently used word "Studies"
7.7. Reviews B and X and the signed non-anonymous review to the MIT Press of my first book used the forceful and infrequently used word "Strengthen"
7.8. Both reviews A, X, and the signed non-anonymous review to the MIT Press of my first book share certain antagonistic words
7.9. Reviews A and X contain numerous textual similarities

7.9.1. Two similarities in section 4 of MLC reviews A and X
7.9.2. Five similarities in section 3 of MLC reviews A and X
7.9.3. Four similarities in section 1 of MLC reviews A and X
7.9.4. Similarity in section 2 of MLC reviews A and X
7.9.5. An additional indication that between MLC reviewer X and may be reviewer A or B

7.10. Similarities between reviewers A, X, and #2

7.10.1. Reviewers A, X, and #2 employ similar words to say they are "extremely suspicious" that genetic programming worked so efficiently
7.10.2 Five similarities between reviewers A, X, and #2

7.11. Similarities between reviewers B and X
7.12. Reviewers A, B, and T2 share the tendency to over-diligently fill in every line of a form with government bureaucrats
7.13. There are only 3 persons among the editors and 32 editorial board members of the Evolutionary Computation journal who are known to be familiar with an lightly attended, oral presentation
7.14. It would be difficult for 2 or more geographically dispersed members of the editorial board of the Evolutionary Computation journal and for 2 geographically dispersed reviewers for the Machine Learning Conference to collude on the reviewing of a particular paper
7.15. Even if 2 or 3 members of the 32 members of the editorial board of the Evolutionary Computation journal were to collude on a particular paper, the telltale electronic >'s would not appear as they did on review document #1
7.16. The transmittal letter from the North American Associate Editor makes the same error as reviewers #1 and #2 of claiming that the paper does not contain a "baseline" comparison even though the paper actually devotes almost 2 pages to this subject
7.17. There are only 2 people among the 24 Machine Learning Conference reviewers and among the editors and 32 editorial board members of the Evolutionary Computation journal who are known to have a close working relationship, at the same institutions, over many years
7.18. An exceedingly close relationship with the editors can be inferred from the unusual hostility of reviewers #1, #2, and #3 of my ECJ paper (thereby raising the question of whether the submitted paper was reviewed entirely inside NRL by non-members of the journal's editorial board)
7.19. An exceedingly close relationship with the ECJ editors can be inferred from reviewer #3's willingness to make a potentially embarrassing unchecked accusation, in writing, that the authors of the submitted paper were not "Up front"
7.20. An exceedingly close relationship with the ECJ editors can be inferred from reviewer #1's willingness to admit, in writing, that he couldn't grasp the main take-home message of the paper
7.21. The absence of normally expected vigilance by the editors of the Evolutionary Computation journal upon receipt of the 3 reviews)
7.22. An institutional affiliation can be inferred from the fact that reviewer #2 took umbrage over an imagined disparagement and slight of classifier systems, LS-1, and SAMUEL
7.23. An institutional affiliation can be inferred from the fact that reviewer #2 took umbrage over an imagined disparagement and slight of LS-2
7.24. An institutional affiliation can be inferred from the fact that reviewer #2 overreached for gratuitous bibliographic citations to work at the Naval Research Laboratory (thereby raising the question of whether the submitted paper was reviewed entirely inside NRL by non-members of the journal's editorial board)

8. Preliminary Conclusions

9. Proposal for binding arbitration, under auspices of the American Arbitration Association, by a retired federal judge of the issues and plagiarism and scientific misconduct

10. Public policy and scientific policy issues involving the scientific peer review process and the Naval Research Laboratory

10.1. Fragility of the scientific peer review process
10.2. Need for established mechanisms for supervision, accountability, and complaint resolution in the scientific peer review process
10.3. Reform of the scientific peer review processat journals and conferences
10.4. Reform at the Naval Research Laboratory

11. References