Ferret-wrangling with the Garden Professors

Expand contents

• The context: my research about drainage layers
• Introducing the Garden Professors
• (Un)scientific criticism
• Arguing to win
• Identifying a cult of expert personality
• How could the story have gone differently?
• Conclusions

I just published a separate article describing a phenomenon I’ve called the cult of expert personality. In summary, it’s something that happens in online communities which have a small number of well-regarded experts or leaders. If those perceived experts start to prioritise praise and superiority over good information and education, their group will turn into a cult of idealising followers who have no way to question or disagree with anything the experts say.

This article is a more detailed description of a recent experience I had which exemplifies that phenomenon.

The context: my research about drainage layers

For a long time, gardeners have sometimes added a layer of rocks or gravel below the soil in a plant pot, with the intention of improving drainage (that is, reducing the amount of water retained in the pot). More recently, some people have started to argue that these drainage layers actually make drainage worse and retain more water.

Despite years of debate, no-one had ever done and published a formal experiment on the matter. All the arguments for and against the effectiveness of drainage layers in pots were theoretical and unproven.

So, I tested it myself. My research paper was recently published in peer-reviewed journal PLOS One. I also wrote a more reader-friendly lay summary of the research here.

In short, my results showed that drainage layers generally reduce the amount of water retained in the pot – in other words, a layer of gravel below the soil does improve drainage after all. This caused quite a stir in some gardening circles where the battle has raged on for years.

Introducing the Garden Professors

The Garden Professors are a small group of horticulture experts who run a blog about garden science, and a facebook group where they answer questions and give gardening advice. They describe themselves as science-based, and their group rules encourage members to share their sources and refer to peer-reviewed evidence. Their blog is one of several online resources which popularised the idea that drainage layers make drainage worse.

I have chosen to name the Garden Professors explicitly in this article because I believe it’s important for readers to be able to verify my descriptions of events, as well as to know which community I am advising wariness around. The Garden Professors’ real-world identities are public, and I am happy to publish this under my own real name too. Their facebook group is accessible to their tens of thousands of readers, but is not publicly visible, so I have anonymised the facebook screenshots I used in this article.

I figured the Garden Professors would be interested to learn about my new research. Their prediction had been tested in a peer-reviewed experiment for the first time ever, and I thought they and their readers would be curious about the results and implications.

I submitted a post to their facebook group with a link to the paper and a summary of the conclusions. The admins immediately declined it with the explanation that the topic had been discussed before, and the conclusions of my research were “incorrect”. This seemed odd since the prior discussions had obviously happened before my research had been published – I wanted to share it because I thought group members would be interested in the new findings.

I contacted the professors directly to invite them to discuss the research. They responded with strange and evasive dismissals, and maintained that they wouldn’t allow me to share the study in their group. But after a few days they made a post of their own, describing and criticising my paper without linking to it directly.

I joined the conversation, and proceeded to have one of the strangest online interactions I’ve ever experienced.

(Un)scientific criticism

Four of the group admins gave their criticisms of the research. Their points were quite unlike anything I would expect from experts in their field. Here are the most memorable, with my explanations and commentary.

~

Criticism: The measurements were taken using units which ignore the effects of gravity and therefore are inaccurate.

It is standard practice to assume that local gravity is constant for experiments conducted on the surface of the Earth. This criticism was so surprising to me that I replied to ask for clarification to make sure I’d understood. Instead of answering, the admin deleted both my question and their original comment.

~

Criticism: The research doesn’t contain numerical data, measurements weren’t conducted on a ratio scale, and there was no statistical analysis.

This comment made me genuinely wonder whether we were looking at the same piece of research.

The paper is full of numbers. There are graphs, textual descriptions, and the complete raw data from every trial are in the supplementary files.

A ratio scale means a continuous numerical measurement with a true “zero” point. Mass and volume, the two measurements used in the paper, are both on a ratio scale.

My research uses a form of statistical analysis called ANOVA post-hoc Tukey tests. This is explained in the method, the relevant p-values are given in the results, and the full statistical calculation tables are included as supplementary files.

~

Criticism: The experiment did not specify the quantity and contents of potting mix, or the amount of water added and drained, and therefore the results are unreliable.

All of these were specified in the paper.

~

Criticism: The research ignores or denies the capillary barrier effect and is therefore incomplete.

The paper covers this effect in detail, and it is the main physical principle which predicts and explains the results.

~

Criticism: The results conflict with existing research and therefore must be wrong.

There has not been any previous research on this situation. Despite being insistent on this point, the professors have not provided a citation for a single peer-reviewed experiment which tested the same situation as mine. The related research that does exist correctly predicts and explains the results.

~

Criticism: The research is published in a multidisciplinary, open-access journal, and is therefore disreputable and wasn’t suitably peer reviewed.

Lack of perceived importance does not mean lack of scientific credibility – research can be valid while not having a dramatic impact on the world.

“Lack of adherence to a scientific field” refers to multidisciplinary research. Some of the most highly-regarded journals in the world are multidisciplinary, for example Science and Nature.

Open access publication is considered the gold standard for making science widely available, and is held to the same rigorous standards as paywalled research. The only difference is that the researcher or publisher covers the costs, so that the final publication is free for anyone to read.

~

All of the experts suggested that the issues they were pointing out made the research entirely meaningless. Healthy skepticism is an important part of the scientific attitude, but outright dismissal is an extreme position that requires proportionally compelling justification.

Their criticisms sound scientific and intelligent on the surface, but they don’t bear scrutiny. At best, the experts seemed to have misread the study or were guessing what was in it. At worst they could have been outright lying in the hope of not being caught out.

Arguing to win

The Garden Professors’ pseudoscientific criticisms weren’t the only thing that made the discussion odd though. The strangest part was the way they conducted the conversation. It felt like trying to engage with a group of slippery ferrets. I’d turn one way to answer a question, only to find whoever I was responding to had gone behind me to do something else, and then suddenly another was getting under my feet before I could get my balance.

Here are some of the things the expert admins did during the fairly short conversation in the facebook comments:

• Deleted both our comments when they said something incorrect and I replied to question it or point it out.
• Never once acknowledged that they had been mistaken when their errors were pointed out.
• Asked me questions, then ignored my answers and switched to criticising something else instead.
• Never once answered any of the clarifying questions I asked about their points or criticisms.
• Removed my comment and gave me an official warning for breaking their rules requiring posts be science-based and non-commercial. The comment was a link to the open access, non-commercial, peer-reviewed research paper. They later posted a link to their own research paper about something else.

I got the distinct sense that their intention was purely to trip me up and dazzle their readers, not to educate others, and especially not to learn anything themselves.

Their behaviour seemed straight out of the Alt-Right Playbook: Never Play Defence. They would make a short quip, I would give a detailed rebuttal, and instead of responding to that they would jump straight to a completely new, snappy statement. That method of arguing is not at all exclusive to the alt-right. It’s a common approach from anyone who wants to look like they’re winning more than they want to be correct.

The Garden Professors purport a science-based approach and encourage reference to published literature. But because my research disagreed with their predictions, they literally censored it to make sure that the only information readers would have about the research was filtered through the professors’ own theatrical criticism.

After the discussion, the professors published a blog post about the research. They left out most of the truly bizarre criticisms they’d made on facebook, but they continued to insist that the result disagrees with existing research and the paper was not sufficiently peer reviewed – while not providing any contradictory studies. (To their credit, they also pointed out some genuine minor limitations to the methods – some that I had identified in the paper and some that I hadn’t.) From all that, they concluded that the study was so flawed as to be utterly worthless and should be dismissed entirely.

I want to make it clear here that this article isn’t about me being upset that my research was criticised. Sincere and constructive criticism is fundamental to the practice of science, and happens extensively before, during, and after publication for every research paper. My study was peer reviewed thoroughly by specialists from PLOS One and from other publications when I submitted earlier drafts. And I’m grateful for that, because it helped me improve my work and made me a better scientist.

My paper isn’t flawless, and it’s certainly not going to cure cancer. But demonstrating that a study is entirely unworthy of acknowledgement requires further peer-reviewed research, not blog posts and facebook comments. I’m fine with my work being evaluated as part of the scientific process – and that is not at all what the Garden Professors were doing.

Identifying a cult of expert personality

So, what were they doing? The Garden Professors’ real identities are public knowledge, and they are all well-established professionals in the field. Between them they have decades of experience in horticulture research and teaching. Why go to all this trouble trying to discredit a pretty unremarkable research paper with a facebook argument?

Let’s look at some of their behaviour through the lens of the descriptors about cult experts.

• The Garden Professors are never wrong. Faced with a new peer-reviewed paper which disagreed with their hypothesis, they immediately dismissed it as incorrect and suppressed any discussion that might have even hinted that their old theoretical prediction was anything less than perfectly accurate. Despite broadly specialising in outdoor and landscape gardening, they did not ask a single genuine question that might have helped them better understand the study or why it differed from their predictions. They remain happy to present their prediction as unquestionably correct in spite of the newly-published evidence which directly disagrees with it.
• The Garden Professors don’t explain or source their information. I sincerely asked several times for clarification about what they meant in their responses, and was ignored every single time. They continue to insist that my study conflicts with decades of research, and continue not to provide a single citation for an existing study which tested the same situation and got different results.
• The Garden Professors don’t encourage others to learn. They mockingly dismissed anyone who took the research seriously, without making any genuine effort to help readers evaluate it for themselves – in fact censoring the study to prevent anyone else from reading it.

• The Garden Professors aren’t bound by their own rules. They removed my comments or gave me official warnings for uncivil conduct when I tried to ask sincere questions, then they mocked and dismissed others with no consequences. They also gave me a warning for posting a link to a peer-reviewed research paper, but allowed themselves to post a link to their own paper.

• The Garden Professors enforce their respect, by mocking, attacking, or silencing people who question them. Like here, when a member thanked me for something I wrote, and then one of the admins replied to insult their reading comprehension.

The Garden Professors collectively behaved in ways that match every descriptor of cult experts.

As in any cult of expert personality, the experts maintain their high status in part through the support and silence of enablers. Although the professors espouse scientific literacy among their members, no-one except the professors made any significant comments about the research itself. Aside from the one polite reply I received above (where they were immediately mocked by the admin), the only other responses from members were effusive thanks and unquestioning agreement with the professors. In a healthier community I would hope to see members feeling safe enough to join in with scientific evaluation.

Members who question or disagree with the experts are dismissed, attacked, or forcibly silenced. All that’s left are those who have fully embraced the professors’ status as utterly infalliable.

It seems clear that over the course of their online existence, through some combination of their inaction and their deliberate choices, the Garden Professors have developed their group into a cult of expert personality.

How could the story have gone differently?

We don’t need to imagine this, because I actually had a parallel experience which nicely demonstrated the contrasts between a cult-like group and a healthier one.

I contacted various potentially-interested parties to discuss my research when it was published. One of those was Robert Pavlis who runs the Garden Myths blog, and an associated facebook group. Just like the Garden Professors, Robert’s blog was another online resource which had spread the word that drainage layers in pots make drainage worse.

When Robert was introduced to my research, he was skeptical because it conflicted with what he believed to be true. Like a reasonable scientist, he took this as a prompt to read the study carefully, to try to find out why this was the case and what could be learned from it. He asked me pointed questions and scrutinised the raw data to make sure he understood how the conclusions followed from the experiment.

If the Garden Professors had done this, they would have been able to understand why my result differed from the previous prediction. The “drainage layers make drainage worse” hypothesis had been largely based on research into soils in the ground. In that context, the addition of a layer of gravel creates more of a barrier to water than the alternative of continuous soil. But my test was based on soil in a pot, where the drainage hole at the bottom is already a barrier to water, and adding a layer of gravel makes it easier to pass through. For those familiar with academic writing, the logic is fairly simple to understand by reading the paper carefully.

The professors didn’t do this, because they were unable to interpret the result rationally. They were too entrenched in their cult expert status and the need to protect their image and ego. Instead they had to take any action available to them to avoid the threat of admitting they were wrong and learning something.

In contrast, once Robert was satisfied that he understood the research, he took the bold step of changing his mind. He was able to accept that there was new information which meant that his previous prediction was no longer accurate. And he was able to admit this to his readers, in the process demonstrating to them how to learn from new science. Robert’s ability to acknowledge new information and admit being wrong makes him a clear example of a community leader who is not a cult expert.

Conclusions

It’s sad to see a community that was clearly well-intentioned has lost its way so badly. The way the Garden Professors manage their group has made it incompatible with their original goal of science-based education. In the end it has become a pseudoscientific cult of expert personality.

I doubt the Garden Professors will read this, because they don’t seem interested in anything I have to say. But if you’re the respected expert of a community and you’re concerned about cult-like behaviour, I don’t think it can ever be too late to start humbling yourself. If you can begin to allow new information in without feeling threatened, and learn to be gracefully wrong, you might just be surprised by what a pleasant environment your group could still become.

If you’re a member of any community that includes highly-regarded experts, make sure to keep asking yourself whether they have really earned that high regard. Remember, no-one should ever be beyond question. No authority should ever be outside of their own rules. If an expert can’t be wrong, you can’t rely on them to be right.