What is Nature Biotechnology good for?

NOTE: Superb piece on the Ermakova-Nature Biotechnology scandal.

EXTRACT: Its owner, Nature Publishing Group no doubt finds that publishing a magazine that does double duty as a science journal and as a trade journal is a highly profitable combination, but equally it is never going to be one that encourages disinterested science
---
---
What is Nature Biotechnology good for?
The case of Irina Ermakova
The Bioscience Resource Project, Dec 4 2007
http://www.bioscienceresource.org/commentaries/brc6.php

Quite likely it surprised many regular readers of Nature Biotechnology that for the September (2007) issue their journal had invented a new article format specifically in order to describe, and then extensively criticise, the work of a researcher that most of them had never heard of before (1). That surprise will only increase if they read the translation, featured on our website, of a Nov 1st article (The excommunication of a heretic) in the Swiss Newspaper WOZ. Readers who thought this new format was simply a curious, if rather aggressive, literary innovation, can now see that this was a story with a disturbing history. Even more interesting however than the ethical shenanigans behind the publication of the interview with Dr Ermakova, is a point not raised by the Swiss newspaper article.

In science, opinions may differ, but it is not usual to attempt to embarrass opponents with overt public criticism. The existence (or imminent prospect) of reproducible data that will settle the issue is usually sufficient to ensure that most disputes never reach the printed page. So why has this dispute followed a different course?

Roundup Ready Soybeans, all of which are derived from a single transgene insertion event (40-3-2), have been on the market for approximately twelve years. They have been grown on millions of hectares and passed regulatory safety assessments in many countries. If a researcher makes a seemingly anomalous finding that RR Soy harms rats then surely all that should be necessary is for their critics to reach for the multitude of studies already in existence for a handy refutation? For RR Soy however, such a body of incontestable data does not exist. It is this remarkable data gap that seems to be behind the Nature Biotechnology interview with Dr Ermakova and quite probably it is for this reason that her unpublished study so alarms the biotech industry.

In the case of RR Soy, there is a single broadly comparable (i.e. multi-generational) study that examines similar endpoints in rodents (in this case mice) fed RR Soy and that also supports the contention that mice are unaffected by RR soy (Brake and Evenson 2004). The problem however is that this is only a single small study and, although published in a peer reviewed journal, it suffers from as many flaws as does the study carried out by Dr Ermakova. For those interested, we can recommend applying the Nature Biotechnology criticisms of Dr Ermakovas’ study, to the Brake and Evenson study.

For example, Dr Ermakova is criticised in the interview for having bought her seeds from ADM Netherlands, even though she says she tested to confirm that they contained the RR transgene. Brake and Evenson in contrast report that they relied on an unnamed seed dealer taking them to a single field of RR soy and a single non-transgenic field, where they obtained soybeans from unspecified cultivars. Brake and Evenson report no attempt to verify the dealers' identification of the soybeans as transgenic or otherwise (Brake and Evenson 2004). It is therefore hard to understand why Brake and Evenson's should be considered a superior method for obtaining samples.

There is one further published study, not mentioned by the Nature Biotechnology critics, that is comparable (in the sense of being multigenerational) to those of Ermakova and Brake and Evenson. It reports histological studies on the offspring of mice fed RR soy (Malatesta et al 2002a). These authors reported ultrastructural alterations to hepatocytes of the offspring of pregnant mice fed RR Soy but no other differences (Malatesta et al 2002a). A further paper from the same laboratory reported biochemical (but not visible) alterations to pancreatic cells of mice fed RR soy for up to 8 months after weaning (Malatesta 2002b). These two papers arguably offer some support for Dr Ermakova's work in that effects of RR soy were observed, although the effects seen were not the same. For example, no effects on offspring body weight or mortality were noted.

Perhaps the most important general point about all these multigenerational studies is that none of them used near-isogenic soybeans grown side-by-side, which is a prerequisite for a properly controlled test of the question that we all want answered: whether the 40-3-2 transformation event can be responsible for altered toxicological or nutritional properties of soybeans.

The critics' challenges to Dr Ermakova's work are mostly reasonable (2). They rest on demonstrating flaws in her methodology and also on the assertion that her work is further contradicted by four mammalian feeding studies of RR soy, even though, unlike Dr Ermakova’s, all of these are single generation studies (Zhu et al 2004; Teshima et al 2000; Cromwell et al 2002; Hammond et al 1996). As in their comparison with the Brake and Evenson work however, the criticisms create the impression that these papers do not themselves suffer from the flaws noted in Dr Ermakova's work. This is not the case however and from a toxicological perspective, the limitations of all these studies (summarised in Tables A and B) are strikingly similar to those pointed out by Dr Ermakova's critics.

Table A details for each study the preparation and selection of soybeans for consumption (e.g. whether RR and control soybeans were grown under the same conditions, whether isogenic lines were used, whether the presence/absence of the transgene was ascertained, etc.) while Table B details key attributes of the feeding studies themselves (such as how many animals were used, what percent soy was included in the diet etc). Looking at Tables A and B it is plain to see that, as a result of their collective limitations (which include short study durations, small numbers of animals and lack of replication), while no adverse effects were reported (though see footnote 3), their individual and collective limitations are highly significant.
[http://www.bioscienceresource.org/docs/BRS-TableA.doc
http://www.bioscienceresource.org/docs/BRS-TableB.doc ]

These inadequacies, which are fundamental to any discussion about whether Dr Ermakova's data are in conflict with the published literature, appear to have been missed entirely by the Nature Biotechnology critics. It is not the only mistake they make however. They seem to have been unaware of the Malatesta papers, they cite Teshima et al and Zhu et al in stating that 'Previous reports in the literature have shown no effects of RR soy on birth weights or pup mortality' (p983) yet neither paper studied birth weights, pregnant rats or pregnant mice. In fact, Teshima et al started their study on rats and mice that were both seven weeks old and Zhu et al started theirs on 28 day old rats. None of these are trivial errors and they make the question - raised by Roland Fischer of WOZ - of why Nature Biotechnology failed to use truly expert referees, a highly pertinent one.

Ultimately, more important than any misrepresentation of the evidence by these four critics, is a question that is central for regulators who are asked to approve and consumers who are offered RR Soy: does the existence of these seven flawed studies tell us anything useful about the safety or otherwise of RR soy? This question is also an interesting one purely from a scientific perspective because current scientific understandings (especially in regulatory science) are very frequently constructed from small numbers of highly imperfect studies. The answer would seem to depend on at least three parameters, all of which are mutually dependent. Firstly, the precise nature of those flaws, because some, such as failing to positively determine the presence of the transgene in the treatments, and equally its absence from the controls, ought to invalidate any experiment regardless of the subsequent quality of data collected (4). The second significant parameter is whether the flaws in each paper are the same or overlapping. Flaws in one paper, and usually these will be data gaps, can sometimes be made up for by the results of another. Lastly, an experiment may be perfectly useful, but nevertheless not support the conclusions which the authors draw. Failure to use isogenic lines and/or to grow them side-by-side means that, whatever the title of a paper may imply, any effect seen in the treatment group cannot be attributed specifically to presence of the RR transgene. Using these criteria, any conclusions about the safety of RR soy based on these data must be extremely limited and highly provisional, and consequently deeply unsatisfactory.

All of which raises an issue for Nature Biotechnology as a scientific journal. The safety or otherwise of RR Soy is a matter of great public health significance. New research suggesting that it might not be safe, especially when the prior research is inconclusive at best, should be a matter of significant concern. Yet Nature Biotechnology saw Dr Ermakovas' work only as an industry threat and publicised her work seemingly only in order to dismiss it. There is no difference at all between what Nature Biotechnology has done to Irina Ermakova and what Fox news did to Marion Nestle when they hired Steven Milloy, then at the Cato Institute, to review her book Food Politics: How the Food Industry Influences Nutrition and Health. And if there is no difference in behaviour between Fox News and Nature Biotechnology, what is the value of Nature Biotechnology as a journal of science?

It seems to us that this interview is part of a pattern and that Nature Biotechnology has for some time been unclear where the line that traditionally separates trade magazines from science journals lies (5). Its owner, Nature Publishing Group no doubt finds that publishing a magazine that does double duty as a science journal and as a trade journal is a highly profitable combination, but equally it is never going to be one that encourages disinterested science (6). Consider what Nature Biotechnology could have done to address the question of transgenic soya biosafety: it could have invited Dr Ermakova to submit her work formally and, whether it was accepted or rejected, it could have written an editorial calling for appropriately controlled high quality independent research to fill the data gaps. It could have pointed out that, if anything the existing data provides hints that there is a need for such experiments. The fact that Nature Biotechnology did none of these should be of deep concern to all its readers.

1) Marshall, A. (2007) Nature Biotechnology 25: 981-98

2) Not all of their points are fair though. For example, Dr Ermakova is criticised for not double-blinding her experiments, yet none of the studies discussed in Nature Biotechnology or in this commentary were double-blinded.

3) Interestingly, though Zhu et al make no mention of it, for all time points tested and for both sexes their results show an almost perfect correlation of decreasing white blood cell counts as RR soya replaces conventional soya in the diet fed to their rats.

4) Although the fact that in some papers the developers of RR Soy were also the experimenters might be considered to mitigate this defect (Table A). Equally however, their collaborators might have tested for their own satisfaction.

5) Compare the treatment of non-target effects of Bt in the review by Romeis et al (2006) in Nature Biotechnology 24: 63-71 with that of Lovei and Arpaia (2005) in Entomologia Experimentalis et Applicata 114: 1-14 or Hilbeck and Schmidt (2006) in Biopesticides International 2: 1-50

6) It is also interesting to note that, unlike most journals, NPG journals have no association with any scientific societies and no independent editorial board.

References
Brake, D. G. and Evenson, D. (2004) Food Chem. Toxicol. 42 29-36
Cromwell, G.L. et al (2002) J. Anim. Sci. 80: 708-715
Hammond, B.G. et al (1996 J. Nutr. 126: 717-727
Malatesta,M. et al (2002a) Cell Struct. Funct. 27: 173-180
Malatesta, M. et al (2002b) J. Anat. 201: 409-446
Teshima, R. et al (2000) J. Food Hyg. Soc. Japan 41: 188-193
Zhu, Y. et al (2004) Arch. Anim. Nutr. 58: 295-310