How Science really works

adam warlock View Profile

Science

20 Aug '09 18:29

adam warlock
Baby Gauss
Ceres
Joined
14 Oct '06
Moves
18375
20 Aug '09 18:291 edit
By Professor Rick Trebino:

[i]The essence of science is reasoned debate. So, if you disagree with
something reported in a scientific paper, you can write a “Comment”
on it.
Yet you don’t see many Comments.
Some believe that this is because journal editors are reluctant to
publish Comments because Comments reveal their mistakes—papers
they shouldn’t have allowed to be published in the first place. Indeed,
scientists often complain that it can be very difficult to publish one.
Fortunately, in this article, I’ll share with you my recent experience
publishing a Comment, so you can, too. There are just a few simple
steps:

1. Read a paper that has a mistake in it.
2. Write and submit a Comment, politely correcting the mistake.
3. Enjoy your Comment in print along with the authors’ equally
polite Reply, basking in the joy of having participated in the
glorious scientific process and of the new friends you’ve made—
the authors whose research you’ve greatly assisted.

Ha ha! You didn’t really believe that, did you? Here’s the actual
sequence of events:

1. Read a paper in the most prestigious journal in your field that
“proves” that your entire life’s work is wrong.

2. Realize that the paper is completely wrong, its conclusions
based entirely on several misconceptions. It also claims that an
approach you showed to be fundamentally impossible is
preferable to one that you pioneered in its place and that
actually works. And among other errors, it also includes a
serious miscalculation—a number wrong by a factor of about
1000—a fact that’s obvious from a glance at the paper’s main
figure.

3. Decide to write a Comment to correct these mistakes—the
option conveniently provided by scientific journals precisely for
such situations.

4. Prepare for the writing of your Comment by searching the
journal for all previous Comments, finding about a dozen in the
last decade.

5. Note that almost all such Comments were two to three pages
long, like the other articles in the journal.

6. Prepare further by writing to the authors of the incorrect paper,
politely asking for important details they neglected to provide in
their paper.

7. Receive no response.

8. Persuade a graduate student to write to the authors of the
incorrect paper, politely asking for the important details they
neglected to provide in their paper.

9. Receive no response.

10. Persuade a colleague to write to the authors of the incorrect
paper, politely asking for the important details they neglected to
provide in their paper.

11. Receive no response.

12. Persuade your colleague to ask a friend to write to the authors
of the incorrect paper, politely asking for the important details
they neglected to provide in their paper.

13. Receive no response.

14. Ask the graduate student to estimate these parameters herself,
and observe that she does a very good job of it, reproducing
their plots very accurately and confirming that the authors were
wrong by a factor of about 1000 and that their conclusions were
also wrong.

15. Write a Comment, politely explaining the authors’
misconceptions and correcting their miscalculation, including
illustrative figures, important equations, and simple
explanations of perhaps how they got it wrong, so others won’t
make the same mistake in the future.

16. Submit your Comment.

17. Wait two weeks.

18. Receive a response from the journal, stating that your Comment
is 2.39 pages long. Unfortunately, Comments can be no more
than 1.00 pages long, so your Comment cannot be considered
until it is shortened to less than 1.00 pages long.

19. Take a look at the journal again, and note that the title, author
list, author addresses, submission date, database codes,
abstract, references, and other administrative text occupy about
half a page, leaving only half a page for actual commenting in
your Comment.

20. Remove all unnecessary quantities such as figures, equations,
and explanations. Also remove mention of some of the authors’
numerous errors, for which there is now no room in your
Comment; the archival literature would simply have to be
content with a few uncorrected falsehoods. Note that your
Comment is now 0.90 pages.

21. Resubmit your Comment.

22. Wait two weeks.

23. Receive a response from the journal, stating that your Comment
is 1.07 pages long. Unfortunately, Comments can be no more
than 1.00 pages long, so your Comment cannot be considered
until it is shortened to less than 1.00 pages long.

24. Write to the journal that, in view of the fact that your Comment
is only ever so slightly long, and that it takes quite a while to
resubmit it on the journal’s confusing and dysfunctional web site,
perhaps it could be sent out for review as is and shortened
slightly to 1.00 pages later.

25. Wait a week.

26. Receive a response from the journal, stating that your Comment
is 1.07 pages long. Unfortunately, Comments can be no more
than 1.00 pages long, so your Comment cannot be considered
until it is shortened to less than 1.00 pages long.

27. Shorten your Comment to 0.80 pages, removing such frivolous
linguistic luxuries as adjectives and adverbs.

28. Resubmit your Comment.

29. Wait three months, during which time, answer questions from
numerous competitors regarding the fraudulence of your life’s
work, why you perpetrated such a scam on the scientific
community, and how you got away with it for so long.

30. Read the latest issue of the journal, particularly enjoying an
especially detailed, figure-filled, equation-laden, and
explanation-rich three-page Comment.

31. Receive the reviews of your Comment.

32. Notice that Reviewer #3 likes your Comment, considers it
important that the incorrect paper’s errors be corrected and
recommends publication of your Comment as is.

33. Notice that Reviewer #2 hates your Comment for taking issue
with such a phenomenal paper, which finally debunked such
terrible work as yours, and insists that your Comment not be
published under any circumstances.

34. Notice that Reviewer #1 doesn’t like it either, but considers that
its short length may have prevented him from understanding it.

35. Also receive the topical editor’s response, pointing out that no
decision can be made at this time, but also kindly suggesting
that you consider expanding your Comment to three pages and
resubmitting it along with your responses to the reviews.

36. Expand your Comment back to three pages, replacing adjectives,
adverbs, figures, equations, explanations, and corrections of
author errors you had had to remove earlier to meet the 1.00-
page limit. And, in an attempt to enlighten Reviewers #1 and
#2, include a separate extended response to their reviews.

37. Resubmit your Comment.

38. Wait three months, during which time, receive condolences from
numerous colleagues regarding the fraudulence of your life’s
work and how sorry they are about it having been debunked.

39. Fail to enjoy your colleagues’ stories of other deluded scientists
in history whose work was also eventually debunked, and try to
explain that, in fact, you feel that you don’t actually have that
much in common with alchemists, astrologers, creationists, and
flat-earthers.

40. Read the latest issue of the journal, which includes another
detailed three-page Comment, almost bursting with colorful and
superfluous adjectives and adverbs, some as many as twenty
letters long.

41. Receive the second set of reviews of your Comment.

42. Notice that Reviewer #3 continues to like your Comment and
continues to recommend its publication.

43. Notice that Reviewer #2 continues to hate it for taking issue
with such a phenomenal paper, which finally debunked such
terrible work as yours, and again insists that your worthless
Comment not be published.

44. Note further that Reviewer #2 now adds that your Comment
should under no circumstances be published until you obtain the
important details from the authors that you confessed in your
response to the reviewers you were not able to obtain and are
not ever going to.

45. Realize that Reviewer #2’s final criticism inevitably dooms your
Comment to oblivion until such time as the authors provide you
with the important details, your best estimate for which is never.

46. Notice, however, that Reviewer #1 now sees your point and
now strongly recommends publication of your Comment. He
also strongly recommends that your Comment remain three
pages long, so that other readers can actually understand what
it is that you’re saying.

47. And, in an absolutely stunning turn of events, note also that
Reviewer #1 writes further that he has also somehow secretly
obtained from the authors the important details they neglected
to provide in their paper and refused to send to you. Even
better, using them, he has actually checked the relevant
calculation. And he finds that the authors are wrong, and you
are correct.

48. Realize that it is now no longer necessary to respond to the
impossible criticism of Reviewer #2, as Reviewer #1 has kindly
done this for you.

49. Add a sentence to your Comment thanking Reviewer #1 for his
heroic efforts in obtaining the authors’ important details and for
confirming your calculations.
adam warlock
Baby Gauss
Ceres
Joined
14 Oct '06
Moves
18375
20 Aug '09 18:302 edits
50. Receive the editor’s decision that your Comment could perhaps
now be published. Unfortunately, Comments can be no more
than 1.00 pages long, so your Comment cannot be considered
further until it is shortened to less than 1.00 pages long.

51. Point out to the editor that most Comments in his journal are
two to three pages long. Furthermore, it was the editor himself
who suggested lengthening it to three pages in the first place.
And Reviewer #1 strongly recommended leaving it that long.

52. Wait a month for a response, during which time, answer
questions from numerous friends regarding the fraudulence of
your life’s work and asking what new field you’re considering
and reminding you of how lucky you are to still have your job.

53. Turn down a friend’s job offer in his brother-in-law’s septic-tank
pumping company.

54. Obtain the latest issue of the journal and enjoy reading yet
another nice lengthy Comment, this one swimming in such
extravagant grammatical constructions as dependent clauses.

55. Receive the editor’s response, apologizing that, unfortunately,
Comments can be no more than 1.00 pages long, so your
Comment cannot be considered further until it is shortened to
less than 1.00 pages long.

56. Download pdf files of all Comments published in the journal in
the past decade, most of which were three pages long. Send
them to the editor, his boss, and his boss’s boss.

57. Receive the editor’s response, apologizing that, unfortunately,
Comments can be no more than 1.00 pages long, so your
Comment cannot be considered further until it is shortened to
less than 1.00 pages long.

58. Shorten your Comment to 0.80 pages, again removing
gratuitous length-increasing luxuries such as figures, equations,
explanations, adjectives, and adverbs. Also again remove your
corrections of some of the authors’ errors.

59. Also, replace extravagant words containing wastefully wide
letters, such as “m” and “w”, with efficient, space-saving words
containing efficient, lean letters, like “i”, “j”, “t”, and “l”. So
what if “global warming” has become “global tilting.”

60. Resubmit your Comment.

61. Wait two weeks.

62. Receive a response from the journal, stating that your Comment
is 1.09 pages long. Unfortunately, Comments can be no more
than 1.00 pages long, so your Comment cannot be considered
further until it is shortened to less than 1.00 pages long.

63. Shorten your Comment by removing such extraneous text as
logical arguments.

64. Also, consider kicking off your coauthor from a different
institution, whose additional address absorbs an entire line of
valuable Comment space. Wonder why you asked him to help
out in the first place.

65. Also, consider performing the necessary legal paperwork to
shorten your last name, which could, as is, extend the author
list to an excessive two lines.

66. Vow that, in the future, you will collaborate only with scientists
with short names (Russians are definitely out).

67. Thank your Chinese grad-student coauthor for having a last
name only two letters long. Make a mental note to include this
important fact in recommendations you will someday write to
her potential employers.

68. Resubmit your Comment.

69. Wait two weeks.

70. Receive a response from the senior editor that you cannot thank
Reviewer #1 for obtaining the missing details and confirming
your results, as this would give the appearance that the journal
was biased in your favor in the Comment review process.

71. Assure the senior editor that, if anyone even considered asking
about this, you would immediately and emphatically confirm
under oath, on a stack of Newton’s Principia Mathematica’s, and
under penalty of torture and death that, in this matter, the
journal was most definitely not biased in your favor in any way,
shape, or form in the current geological epoch or any other and
in this universe or any other, whether real or imagined.

72. Receive a response from the senior editor that you cannot thank
Reviewer #1 for obtaining the missing details and confirming
your results, as this would give the appearance that the journal
was biased in your favor in the Comment review process.

73. Remove mention of Reviewer #1’s having obtained the
necessary details from the acknowledgment, realizing that it’s
probably for the best in the end. If word were to get out that,
in order to do so, he had managed to infiltrate the allegedly
impenetrable ultrahigh-level security of the top-secret United
States government nuclear-weapons lab, where it happens that
the authors worked, he would likely be prosecuted by the
George W. Bush administration for treason. And if he’s
anything like the other scientists you know, he probably
wouldn’t last long in Gitmo.

74. Resubmit your Comment.

75. Wait two weeks.

76. Receive a response from the journal stating that, in your
submitted MS Word file, the references are not double-spaced.
Your Comment cannot be considered for publication until the
references in this document are double-spaced.

77. Add lines between the several references, a process that
requires a total of twelve seconds.

78. Resubmit your Comment, a process that, due to dysfunctional
journal web-site problems, requires a total of three hours.

79. Wait two weeks.

80. Receive a response from the senior editor that, while your
Comment is now short enough and properly formatted, over the
many modifications and shortenings that have occurred, its tone
has become somewhat harsh. For example, a sentence that
originally read, “The authors appear to have perhaps
accidentally utilized an array size that was somewhat
disproportionate for the corresponding and relevant waveform
complexity,” has evolved into: “The authors are wrong.”

81. Have numerous telephone conversations with the senior editor,
in which you overwhelm him with the numerous other issues
you have had to deal with during the Comment evaluation
process until he forgets about your Comment’s tone. Indeed,
compared to your verbal tone during these telephone calls, the
paper’s tone seems downright friendly.

82. Celebrate this minor victory by deciding not to include in the
final draft of the Comment’s Acknowledgments section a
description of certain individuals you’ve encountered during this
process—a description that would have involved such colorful
terms as “bonehead” and “cheese-weenie.”

83. Wonder whether your Comment has finally been sent to the
authors for their Reply, or instead was lost, trashed, or sent
back to the reviewers for further review and possible rejection.

84. Wait four months, during which time, respond to numerous
close relatives regarding the fraudulence of your life’s work and
who remind you that at least you still have your health, albeit in
a noticeably deteriorating state over the past few months. And
perhaps you’d like to join them at the local bar for its daily
Happy Hour.

85. Take them up on their offer, but learn that they expect you to
pay for drinks, which, regrettably, you can’t because sales at
the small company you formed to sell devices based on your
work have fallen to essentially zero.

86. Learn from one of your grad students that a potential employer
asked her, “Hasn’t your work recently been discredited?”

87. Learn that she was not granted an interview.

88. Attend a conference, where a colleague informs you that he is
Reviewer #1. Attempt to hug him, but be advised that a simple
“thank you” for merely doing his job is sufficient.

89. Learn from Reviewer #1 that he has not received the authors’
Reply for review, or any other correspondence from the journal
in the several months since he submitted his review.

90. Realize that you had stopped carefully reading the journal, and,
as a result, had missed the “Erratum” published by the authors
on the paper in question six months earlier, shortly after you
submitted your short-lived three-page version of the Comment.

91. Note that, in this “Erratum,” the authors actually admitted no
errors and instead reported new—similarly incorrect—numbers,
which they concluded “do not change any conclusions” in their
original paper.

92. Feel old, as you can remember the days when Errata involved
correcting old errors and not introducing new ones.

93. Note also that, in their “Erratum,” the authors have actually
responded to some highly specific criticisms of their errors you
mentioned in the three-page version of your Comment—
criticisms that you had removed when shortening it to meet the
journal’s strict 1.00-page limit. Criticisms the authors couldn’t
possibly have known about in view of the journal’s strict
confidentiality rules for submitted papers, unless this version of
your Comment was somehow leaked to them...

94. Realize that, with this “Erratum,” the authors have effectively
already published their “Reply” to your Comment.

95. Note also that, while your Comment has been kicking around for
close to a year, its publication date nowhere in sight, the
authors’ “Erratum” was published in a mere nineteen days.

96. With two mathematical mistakes by the authors to consider now
and plenty of time in which to consider them, realize that their
main mathematical error was simply to forget to take the
square root when computing the “root-mean-square”—a childish
mistake.
adam warlock
Baby Gauss
Ceres
Joined
14 Oct '06
Moves
18375
20 Aug '09 18:31
97. Note that this is consistent with the fact that, on both their
paper and “Erratum,” one of the authors’ names is misspelled.
This is consistent with the fact that, by now, you’ve already
spent approximately 100 times as much time correcting their
errors than they spent making them.

98. Realize that you must now modify your Comment to also include
a discussion of the “Erratum.” Ask the editor if you can do this.

99. Receive a response from the editor that, after much discussion
among the journal editors, it has been decided that, yes, you
can do this.

100. Include a couple of short sentences debunking the “Erratum” in
your Comment, using up two valuable lines of text and three
valuable lines in the reference list due to its rather long title.

101. Realize that your Comment is now several lines longer than the
do-or-die 1.00-page limit.

102. Shorten your Comment by omitting noncritical words like “a,”
“an,” and “the,” giving your Comment exotic foreign feel.

103. Also, take advantage of the fact that, in some literary circles,
sentence fragments are considered acceptable. Decide that,
indeed, verbs are highly over-rated.

104. Declare “death to all commas”—a worthless piece of
unnecessary punctuation if ever there was one.

105. Consider using txt msg shorthand 4 actual words 2 further
shorten ur Comment, but decide not 2 when u realize that the
hundreds of frowny-face emoticons u couldn’t resist adding
actually lengthened ur Comment 2 2 pages 🙁

106. Resubmit your Comment.

107. Realize that modifying your Comment to include the “Erratum”
has now, unfortunately, opened it up for additional criticism
from the editors and possibly the reviewers.

108. Receive a phone call from the senior editor, who takes
advantage of this opportunity. He has suddenly remembered
that your Comment’s tone is a bit harsh. He is concerned that
the authors, who appear to be highly motivated and quite crafty,
will complain loudly and aggressively about the obviously
preferential treatment your Comment is clearly receiving from
the journal and make his life miserable. He objects to nearly
every sentence in your Comment, in each case, insisting on a
considerably longer sentence. He insists that you not say that
the authors are “wrong” and suggests instead “perhaps
mistaken.” He also insists on replacing the word “so” with the
unforgivably long “therefore.”

109. Realize that, if you accede to his demands, your Comment will
be an unacceptable 1.2 pages long, dooming your Comment to
oblivion.

110. Also learn from the senior editor that you cannot thank
Reviewer #1 even for simply “confirming your calculations,” as
this would also reveal the obvious preferential treatment your
Comment has clearly received from the journal.

111. Explain that this is a common type of acknowledgment,
revealing no preferential treatment by the journal whatsoever,
and send him a copy of a recent paper from his journal in which
the authors thank a reviewer for actually proving several
theorems for them.

112. Learn from the senior editor that another reason that you
cannot thank Reviewer #1 is that there is no record of Reviewer
#1 actually having confirmed your calculations. Apparently, the
paper on which it was printed has, over the eons, turned to dust.

113. Send a copy of the email from the journal containing Reviewer
#1’s review to the senior editor.

114. Also, offer to put the senior editor in touch with Reviewer #1, in
case all records of Reviewer #1’s identity have also been lost.

115. Also, learn from the senior editor that he admits no expertise in
your field but that he will nevertheless not allow you to say in
your Comment that the approach that you proved twenty years
earlier is “fundamentally impossible” is “fundamentally
impossible.” Instead, you must say that it “has not been shown
to be possible.”

116. Note that, if this could accurately be said about perpetual-
motion machines, it would rekindle interest in that long
forgotten field.

117. Receive no response.

118. Realize that this is probably good news.

119. Encounter a journal representative at a conference, who kindly
mentions that the one-page version of your Comment was, in
fact, sent to the authors for their Reply. And, after a series of
delays, they have submitted it. But, unfortunately, it is extremely
contentious and will be rejected unless toned down significantly.
It’s as if, for some reason, they want it to be rejected.

120. In preparation for the final phase of the Comment process, write
to the editor asking if you will be able to see the Reply to your
Comment and make minor modifications in view of it, as allowed
by most journals.

121. For once, obtain a quick response: “No.”

122. Finally receive notice from the editor that the authors’ official
Reply to your Comment has been reviewed and processed.
Unfortunately, it was not found suitable for publication and so
was rejected. And because, for maximum reader enjoyment, it
is the policy of this journal that a Comment cannot be published
without a Reply, your Comment cannot be published. This
decision is final.

123. Be advised that the journal thanks you for submitting your
Comment, and you should feel free to submit a paper on a
different subject in the future, as this journal features the most
rapid publication of any journal in this field.

Addendum: This ridiculous scenario actually occurred as written; I
didn’t make it up. I confess that, of course, I exaggerated the
responses from competitors, colleagues, friends, relatives, and myself,
but not those of the journal editors or the authors. Those events all
happened exactly as I’ve described them.
The fate described in the last two steps actually occurred to a
different Comment, which I submitted to a different journal a few
years earlier, and which, in fact, never was published, precisely for the
absurd reason given.
Over a year after submitting the Comment discussed in all the other
steps, realizing that it was clearly doomed to oblivion, I sent a copy of
this story to the senior editor’s boss. Shortly afterward, I received a
call from the senior editor, who had suddenly withdrawn all of his
objections. The Comment was fine as it was, and it would be
published!
However, I was still not allowed to see the authors’ Reply until it
was actually in print. And when it appeared, it reiterated the same
erroneous claims and numbers (for the third time!) and then
introduced a few new erroneous claims, which, of course, I am not
allowed to respond to. So I’ve simply given up.
I’ve withheld the names of the various individuals in this story
because my purpose is not to make accusations (as much as I would
like to; they’re certainly deserved), but instead to effect some social
change. Nearly everyone I’ve encountered who has written a
Comment has found the system to be heavily biased against well-
intentioned correcting of errors—often serious ones—in the archival
literature. I find this quite disturbing.
And would it have killed these authors to email me their “results”
prior to publishing them, so I could’ve enlightened them before they
committed themselves to their errors in print, thus avoiding all this
pain?
Finally, I should also mention that, to keep this story light and at
least somewhat entertaining, I actually simplified it somewhat,
omitting numerous additional steps involving journal web-site crashes,
undelivered emails, unreturned phone calls to dysfunctional pagers,
complaints to higher levels of journal management, and some rather
disturbing (and decidedly unfunny) behavior by the authors and
certain editors.

After all, I wouldn’t want to discourage you from submitting a
Comment.
Melanerpes
Joined
08 Oct '08
Moves
5542
21 Aug '09 20:483 edits
Sounds like this involved an issue that people have strong feelings about (global warming comes to mind). And it appears the editor was using every rule he could think of to avoid printing the comment -- if there really was a 1.00 page limit for comments, then why did it only apply to your comment?

Another possibility is that this prestigious journal did not want to have to deal with the embarassment of publishing a study that was so obviously flawed - and your comment would have exposed everything.

Perhaps you should have gone to a rival journal - that probably would have been very eager to use your results (and your story) to bring holier-than-thou journal back into the earthly realm.

Perhaps there needs to be a professional journal (meta-journal?) devoted to discussing the procedures used by various journals. A place where you could produce your whole tale for the entire scientific community to see.
divegeester
Joined
16 Feb '08
Moves
116715
22 Aug '09 09:32
Good grief!

Levels of incompetence like this always make me suspicious of other motivations. I suspect political interference, especially if global warming was the topic. How else will they get us to buy into all those bloody ridiculous wind turbines!

Good luck.
adam warlock
Baby Gauss
Ceres
Joined
14 Oct '06
Moves
18375
22 Aug '09 11:27
I think I gave the wrong impression. This didn't happen to me but to professor Rick Trebino: http://www.physics.gatech.edu/people/faculty/rtrebino.html

This type of things happens a lot in academia but doesn't get talked about much.
Melanerpes
Joined
08 Oct '08
Moves
5542
26 Aug '09 12:481 edit
In college, I took a class on the "sociology of science" and the experiences of professor Trebino would fit right in. Studies that challenge a prevailing value system have a much harder time gaining acceptance - while studies supporting those beliefs might escape rigorous questioning.

There's a tendency to think that science occurs by itself - when obviously, it involves a community of human beings - and those people are subject to the same social and political forces as everyone else.
twhitehead
Cape Town
Joined
14 Apr '05
Moves
52945
27 Aug '09 08:49
Originally posted by Melanerpes
In college, I took a class on the "sociology of science" and the experiences of professor Trebino would fit right in. Studies that challenge a prevailing value system have a much harder time gaining acceptance - while studies supporting those beliefs might escape rigorous questioning.
According to the story, the professors lifes work was challenged by an article. Surely it was the professors point of view that was the 'prevailing value system'?
I also found it interesting that he chose to publish his findings only through a comment, surely he must have at some point published is findings through other channels?
Further, a large part of the story including things like the 1 page limit seem to have been due to poor design of the magazines website or failure of the professor to use it or realize its limitations. He seems to have struggled constantly with the website but never thought to simply bypass it. I find it highly unlikely that the magazine only accepts comments via the website and he seems to have been in regular communication with the editors yet he repeatedly went back to a dysfunctional website.

I would have tried to get round the 1 page limit by splitting my comment into multiple comments.
Bosse de Nage
Zellulärer Automat
Spiel des Lebens
Joined
27 Jan '05
Moves
90892
27 Aug '09 09:02
Originally posted by Melanerpes

There's a tendency to think that science occurs by itself - when obviously, it involves a community of human beings - and those people are subject to the same social and political forces as everyone else.
Has this insight not been factored into the philosophy of science somewhere?

Science Forum

How Science really works