NRLgate -
Plagiarism by Peer Reviewers

Sections 2.8 thru 2.16

This page is part of the NRLgate Web site presenting evidence of plagiarism among scientific peer reviewers involving 9 different peer review documents of 4 different journal and conference papers in the fields of evolutionary computation and machine learning.

This page contains sections 2.8 through 2.16 of "Evidence of plagiarism in reviews A and B of a paper on empirical discovery submitted to the Machine Learning Conference."

Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page

2.8. Both reviewers A and B both neglected section 5 of the paper review form

Reviewer A's entire response to section 5 (entitled "General comments for the author(s)") for my MLC paper on empirical discovery is as follows:
See above.
Reviewer B's entire response the same section of the paper review form for this paper is as follows:
Same comments as above.

(Emphasis added).
Of course, scientific peer reviewers sometimes neglect a section of their paper review forms. Two reviewers of the same paper may sometimes both decide skip some section of their paper review forms. Two reviewers of the same paper may sometimes both decide to skip the very same section of their paper review forms.

How common is it for peer reviewers not to offer any suggestions to the author on their paper review forms?

I was the general chair of the Genetic Programming 1996 Conference and have access to a computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the GP-96 conference.

Section 8 (entitled "Suggestions to Author") on the 16-part GP-96 paper review form was entirely blank on only 5% (19 of 316) of the review documents. An additional 4% (14 of 316) of the reviews did not provide any advice to the author in section 8, but nonetheless over-diligently filled in the blank line of section 8 with a vacuous phrase (e.g., "see above"). That is, 91% of contemporary peer reviewers from this field of science affirmatively offered some suggestions to the author.


2.9. Both reviewers A and B employed the same word while neglecting section 5 of the paper review form

Many different words can be used to express the same idea --- even the idea of saying nothing.

As previously mentioned, 4% (14 of 316) of the reviews did not provide any advice to the author in section 8 of the GP-96 paper review form, but nonetheless over-diligently filled in the blank line of section 8 with a vacuous phrase. The frequency of various vacuous phrases included the following:
(Emphasis added).
Thus, the chance is only 2 in 316 that a reviewer would not give the author any advice, but nonetheless over-diligently fill in the blank on the paper review form with a phrase incorporating the word "above."

This is not the only occasion when we encounter such over-diligence in filling in every blank line on a form. See section 5.7. See section 7.12.

2.10. The opening sentences of both reviews A and B gratuitously provided the same unrequested information

Question 1 of the MLC paper review form asks the reviewer to evaluate the "significance" of the work described in the submitted paper.
Significance: How important is the work reported? Does it attack an important / difficult problem or a peripheral / simple one? Does the approach offer an advance.

(Emphasis added).
Reviewer A begins his review,
This paper reorts on a technique of learning concepts expressed as LISP expression using genetic algorithms. This is a topic of general interest.

(Spelling error of "reorts" in original).
(Emphasis added).
Reviewer B's begins his review,
The author describes an interesting set of results using GAs for empirical discovery and concept formation.

(Emphasis added).
Notice how the opening sentences of both reviews ignored the question concerning "significance" that was actually posed by paper review form. Instead, both of these opening sentences were unresponsive to the question being asked. Both reviews began by gratuitously providing an unrequested summary of the subject matter of the paper.

Is it commonplace for peer reviewers in this field of science to gratuitously and unresponsively begin their answer to a paper review form's question on significance on the paper with an unrequested summary of the paper?

We again make reference to the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference.

Question 2 on the paper review form used by the GP-96 conference asked about the "significance" of the paper.
Significance of the Problem: Is the subject of this paper important?.
98.4% (311) of these 316 review documents began by addressing the question that was actually asked by the paper review form. That is, they said something related to whether the work described in the paper was scientifically important.

Only 1.6% (5) of these 316 review documents began by gratuitously and unresponsively providing an unrequested summary of the subject matter of the paper.

Both reviewers A and B of my MLC paper on optimal control strategies appear to belong to a small minority of reviewers. Isn't it improbable that two independent-acting reviewers of the same paper would both belong this particular small minority?

It would seem unlikely to draw 2 peer reviewers with this particular unusual behavior on the same submitted paper. However, if one peer reviewer were writing his review with the already written review of another review of the same paper in front of him, the suggestive power of the wording of the first review might cause him to pattern his opening sentence after the first reviewer's opening sentence.

This is not the only occasion when we encounter opening sentences of two reviews of the same paper that unresponsively provide a summary of the submitted paper instead of providing the requested evaluation of its significance. As will been seen elsewhere, both reviewers X and Y of my MLC paper on optimal control strategies belong to this same small minority. See section 3.9.

2.11. Both reviewers X and Y substituted "Genetic Algorithms" or "GA" in lieu of the author's chosen term

Reviewer A entire response to section 1 for this paper on genetic programming (GP) follows:
This paper reorts on a technique of learning concepts expressed as LISP expression using genetic algorithms. This is a topic of general interest. The methodology adopted prevents a clear assessment of how much over advance this approach represents.

(Spelling error of "reorts" and grammatical error of "over advance" in original).
(Emphasis added).
Reviewer B's entire response to section 1 (entitled "significance") for this paper on genetic programming (GP) follows:
The author describes an interesting set of results using GAs for empirical discovery and concept formation.
(Emphasis added).
Notice that both reviewers A and B imposed this substitution of "genetic algorithm" or "GA" for the author's chosen term (perhaps offensive to both) that actually appears in the submitted paper.
This is not the only occasion when we encounter two peer reviewers of the same paper both belonging to the small minority of peer reviewers who impose their own term in this manner in lieu of the author's chosen term. As will been seen elsewhere, both reviewers X and Y of my MLC paper on optimal control strategies and reviewer T2 of the submitted TAI paper have this same propensity. See section 3.10. See section 5.4.


2.12. Lock-step treatment by both reviewers A and B in section 2 of the paper review form

The joint usage of the same noun ("approach") at the beginning of these sentences is yet another similarity between the two reviews, as is the agreement between the two reviewers that the application described in the paper is "original" ("new").

Reviewer B's entire response to section 2 (entitled "Originality") of the paper review form is as follows:
The approach is original and quite interesting.
(Emphasis added).
Reviewer A's entire response to this section entitled "Originality" for my MLC paper on empirical discovery is as follows:
The approach has been reported on previously in MLW89. The applications here are new. (Emphasis added).

2.13. Both reviewers A and B used a similar hostile and antagonistic tone

Reviewers of scientific articles generally maintain some level of civility in their written reviews (even when they are making negative comments and judgments about a paper). Of course, a small minority of peer reviews drift over the line. Both reviewers A and B seem to belong to this minority.

Reviewer A of my MLC paper on empirical discovery says,
This is extremely suspicious ...
Reviewer A says,
(who cares?)
Reviewer A says,
results are claimed to appear
Reviewer A says,
The presentation suffers from an abundance of irrelevant details
Reviewer B of my MLC paper on empirical discovery says,
The paper is annoyingly spotty.
Reviewer B also says,
The reader is left with the possible interpretation that, after weeks of trying, the GA managed to get it right once!

Keep in mind that the total size of review A is only 218 words and review B is only 138 words.

How common is it for peer reviewers to be hostile in this way?

We again make reference to the computer file containing the 64,109 words of the 316 paper review forms from the 86 peer reviewers of the genetic programming papers at the Genetic Programming 1996 Conference. All of the above highlighted words or phrases are almost always pejorative in the context of a review (as they indeed were in the reviews in the GP-96 computer file in which they appeared). The frequency of occurrence of the following words is shown below: Thus, both reviewers A and B of my MLC paper on optimal control strategies appear to belong the small minority (about 2%) of peer reviewers with these hostile words.

It would seem unlikely to draw 2 peer reviewers with this particular hostile tone on one particular submitted paper. However, if one peer reviewer were patterning his review after the words and tone established by an already written review of the same paper, the probability of a hostile tone migrating from one review to another is no longer a low percentage.

This is not the only occasion when we encounter two peer reviewers of the same paper both belonging to the small minority of peer reviewers who have a hostile tone. Reviewers #1, #2, and #3 of my paper submitted to the Evolutionary Computation journal seem to belong to this same minority. See section 4.8. See Section 7.8.


2.14. Both reviewers A and B offered similar advice about statistics, using the same trio of phrases

Reviewer A also says,
The paper suffers from a lack of data.
...
The interesting data concerns the general form of learning curves over a number of runs.

In order to judge, it would be necessary to see the results compared against an alternative search technquie, perhaps even random search.

(Grammatical error of "authors" in original).
(Emphasis added).
Reviewer B says,
Rather, he needs to give us some graphs, charts, data, etc. that indicate how the GAs perform on-the-average on these problems. The reader is left with the possible interpretation that, after weeks of trying, the GA managed to get it right once!

(Emphasis added).
In terms of semantics, reviewers A and B cover 3 points, in the same order, but using different words.

First, reviewer A complains about
a lack of data
and reviewer B complains about
he needs to give us some graphs, charts, data
Second, review A wants
learning curves over a number of runs
while reviewer B wants
graphs, charts, data, etc. that indicate how the GAs perform on-the-average
Third, reviewer A talks about
random search
while reviewer B
The reader is left with the possible interpretation that, after weeks of trying, the GA managed to get it right once!
The issue here is not, of course, whether the paper genuinely needed any such "data," but that two reviewers zeroed in on the same defect in the space of two very short reviews (218 and 138 words, respectively) using the same trio of words. It has been my experience (and probably the reader's own experience concerning reviews of his own papers) that even when a submitted paper has a clear deficiency, it is uncommon for two reviewers (out of two) to specifically mention that particular deficiency. Independent reviewers almost always make very different comments, in different words, and in different ways --- particularly in short reviews.

2.14.1 The joint advice of reviewers A and B was at variance with the prevailing practice concerning empirical discovery


Now let's discuss whether the question of whether this submitted paper genuinely needed "on-the-average" "data" "over a number of runs." Would most contemporary members of the machine learning community would agree with reviewers A and B that this particular paper on empirical discovery and concept formation needed "on-the-average" "data" "over a number of runs."

The two simpler problems in my Machine Learning Conference paper were benchmark problems taken directly from well-known and well-regarded works in the machine learning community. Neither of these well-known papers contained any "on-the-average" "data" "over a number of runs" for their respective problems. One problem treated in my MLC paper was the most difficult problem contained in the only existing book-length treatment of empirical discovery (Langley et al. 1987). That 357-page book did not contain any such "on-the-average" "data" "over a number of runs" for any of the several dozen problems in the book. A second problem in my submitted MLC paper was harder than any of the problems in Langley's book and was chosen because the methods in Langley's book would not work on it. Pat Langley is a leading figure in the machine learning community. Pat Langley was the founding editor of the Machine Learning journal and has written editorials on what good experimental papers on machine learning should contain. A third problem treated in my MLC paper came right out of a widely-cited paper by Quinlan on decision trees (1986). Quinlan's paper did not contain any such "on-the-average" "data" "over a number of runs" for his particular problem. Ross Quinlan is a leading figure in the machine learning community. The fact is that for the particular field of empirical discovery and concept formation covered in my MLC paper on empirical discovery, prevailing practice did not call for such "on-the-average" "data." Needless to say, neither Langley, Quinlan, nor I are actively opposed to collecting and reporting such averages. However, unlike some other areas of machine learning research, the primary question that researchers usually ask is whether they can find any satisfactory model for the given observed data (and, usually, whether it generalizes to other previously unseen examples of the data). The question of the average probability of emergence of a suitable empirical model over a number of runs is secondary. Indeed, when one is doing empirical discovery, one is usually very satisfied to find any one suitable model. The frequency of emergence is a perfectly legitimate concern, but it is secondary to the main issue in empirical discovery. The even more important point is that such averages were not contained in the well known benchmark sources from which I obtained these 2 problems. I think that, for the particular fields of empirical discovery and concept formation covered in this particular paper on empirical discovery, reviewer A and B's joint complaint is a minority point-of-view in the machine learning community. (It would not, however, probably be a minority point of view in the genetic algorithm community, where collection of such "on the average" "data" is far more prevalent). Of course, the important point here (in this discussion of plagiarism) is not whether minority point-of-view is right or wrong. The important point is that both reviewers raised this same complaint in their very short reviews and that their joint point-of-view would not likely be shared by other members of the machine learning community for the particular problems in the submitted paper. It would seem unlikely to draw 2 peer reviewers at the Machine Learning Conference with this particular point-of-view on one particular submitted paper.

2.15. Reviewers A and B both ignored the intellectual content of the submitted paper

Both reviews A and B are notable in that they completely avoided any discussion of the substance of my MLC paper on empirical discovery and concept formation. The entire focus of the comments was directed to aspects of the paper that could be obtained by merely browsing the paper at a high level.

What was the substance of my MLC paper?

First, it presented a new way (using genetic programming) to create decision trees. The paper demonstrated the effectiveness of the new technique by solving the same example problem treated in Quinlan's well-regarded and widely cited article on decision trees (1986).

Second, the paper presented a new way (using genetic programming) to build models from empirical data (data-mining). It solved the hardest problem (the Kepler's law problem) contained in the only book-length treatment of empirical discovery existing at the time (Langley et al. 1986).

Third, the paper then went on to solve a problem that was harder than any in the book of Langley et al 1986 and that clearly could not be solved by the technique of that book, namely a problem involving finding a mathematical relationship between the rate of inflation in an economy and other economic data.

Thus, the submitted 12-page paper offered a new way (i.e., genetic programming) to tackle existing recognized problems in two recognized area of machine learning.

There is nothing in either review A or B that deals with the actual substance or subject matter of my paper on empirical discovery and concept formation.

2.16. Reviewer B's quotation of "on one run ..." is inaccurate --- thereby suggesting the time sequence of the plagiarism

The joint action of reviewers A and B in elevating the same unmemorable 3-Word prepositional phrase "In one run, ..." to quotation marks, their joint grammatical error in using the ellipsis, and their joint choice of the same section of their paper review forms to place this quotation together indicate that the second reviewer had the exact text of the first review in front of him when he wrote his review.

This modus operandi of the collusion here was not of "cutting and pasting" entire sentences of one review in order to construct a second review. Instead, the second reviewer reworded (and freshly typed) the first review --- using the already written review as a template for thoughts, choice of words, punctuation, grammar, grammatical errors, and placement of items within his paper review form.

The transformation of each already written review into its slightly altered form required the expenditure of some effort (albeit somewhat mechanical). The transformation of one already written review into its paraphrased form required conscious effort expended by a person who knew he was acting improperly at the time he did it. Of course, the transmission of an already written review to a second peer reviewer was also a conscious activity executed by a person who knew he was acting improperly at the time he did it.

The same modus operandi of collusion was employed by reviewers A and B of my MLC paper on empirical discovery, by reviewers X and Y of my MLC paper on optimal control strategies, reviewers #1, #2, and #3 of my paper submitted to the Evolutionary Computation journal, and reviewers T2 and T3 of the paper submitted to the Tools for Artificial Intelligence conference.

The fundamental purpose of quotation marks is to capture words that are so significant that only the author's precise words do justice to the important idea involved. When writers quote such memorable words, they usually carefully check to be sure that they have correctly transcribed the quoted words.

But, notice the 3 words inside reviewer B's quotation marks for my MLC paper on empirical discovery:
"on one run ... "

(Quotation marks and ellipsis in the original).
(Emphasis added)
Where did reviewer B get these 3 words?
These 3 words do not appear anywhere in my submitted 3,118-word MLC paper on empirical discovery!
In fact, the only similar phrase appearing anywhere in my paper is as follows (as accurately quoted in review A):
"In one run, ..."

(Quotation marks and ellipsis in the original).
(Emphasis added)
Reviewer B's error in changing "in" to "on" suggests the sequential order in which the plagiarism took place. Reviewer A actually extracted the 3 words "In one run" in my submitted MLC paper. For whatever reason, reviewer A decided that these 3 words were worthy of quotation marks. Reviewer A then made the grammatical error of using the ellipsis. Reviewer B had a copy of reviewer A's already-written review in front of him (either on paper, on a computer screen, or via e-mail) and embarked on a process of slightly reworking and altering A's words so that his review would look different from review A. In the process of hastily paraphrasing and mechanically tweaking of reviewer A's words, reviewer B carelessly plowed right past the onset of the quotation marks and continued his tweaking. Reviewer B plowed past the quotation marks because he was not the person who originally decided that the quoted works were memorable. Reviewer B was merely engaged in a thoughtless and mechanical task of changing words around in order to make his review look different from review A. In his mechanical haste, reviewer B made his review look different from the submitted MLC paper --- instead of different from review document A.

This error establishes the order of the plagiarism. If reviewer B had been the one who actually read my paper and made the original error of converting "in" to "on," then review A would also have contained the erroneous word "on."

Review B is not the only occasion when we received a peer review containing the phrase,
"on one run ... "

(Quotation marks and ellipsis in the original).
(Emphasis added)

As will be seen later, a 31-word review (T2) of an entirely different paper submitted to a different (but contemporaneous) scientific conference also contained
"on one run ... "

(Quotation marks and ellipsis in the original).
(Emphasis added)

Review T1 contained the phrase"on one run" even though this phrase "on one run" did not appear in the paper submitted to this second conference. This error by reviewer T2 for the TAI conference will be discussed later because it points to the identity of the plagiarizers (since only 2 persons were reviewers for both the MLC and TAI conferences and since these same 2 persons are the only persons from the MLC and TAI conferences who are also among the editors and editorial board of the Evolutionary Computation journal). See section 5 through section 5.2.

Author: John R. Koza
E-Mail: NRLgate@cris.com
Go to top of NRlgate Home Page
Go to Abbreviated Table of Contents
Go to Detailed Table of Contents
Go to Complaint Letter to the Evolutionary Computation journal
Go to top of Previous Page
Go to top of Next Page