The Importance of Peer-Review, or How to Spot a Huckster.

Hucksters are individuals who claims to have all of the answers; that “main stream” industries are corrupt profiteers out to harm you or your pet’s health for financial gain; and that only be listening to them (the Huckster) and/or by buying their products will you and your pet live long happy and healthy lives. So basically everyone else is lying/conspiring except for them. Sounds a bit to me like Kaa from the Jungle Book as he tries to wrap Mowgli up for dinner, “trussssst in me…”.

How I feel about Hucksters…

Even if you believe that large corporation (be it food or pharmaceutical companies for pets or people) are solely driven by profit, how much long-term profit is there in harming your customer?  Small corporations become large corporation because they provide a service or product that people (the ones with the wallets) want to purchase. There are numerous business books written on this very topic. Corporations with any financial and business sense want those wallet holders buying products for as long as physically possible. Their financial incentive is to prolong health and longevity not shorten it.

In the world of pet food and pet care, there is no shortage of people claiming to have all of the answers and a “study” that was recently “published” online that was too good (and by “good” I mean really bad) not to weigh in on.

Huckster Red Flag: Presenting “findings” through media outlets instead of peer-reviewed journals.

It is important to understand that anyone can write a book or develop a website on any topic they wish. There is no requirement that they need to be an expert in a particular field (or even have any training in a particular field) to voice their opinions loudly and publicly. When I started my consulting website and this blog I didn’t have to prove my credentials before I started. I paid my website registration fee (website) or filled in a registration form (blog) and off I went. I told you my credentials and experience and many people will take me at my word and never look further, but I could easily have made it all up. Fortunately, my education and experience are easily verified as are those of my Veterinary Nutrition colleagues, yet self-taught and self-proclaimed nutrition “experts” still seem to hold more sway than actual experts when it comes to matters of pet food and supplements. Forget the scientific process, these individuals want you to believe them because they say so.

This recent posting from The Spudd (thespudd.com) sums up my take on pseudosciences.

Where I think opinion starts to cross the line into pseudoscience is when experts (real or imagined) wrap their opinion in charts and science-y sounding words in an attempt elicit maximum emotional response, but without providing any actual results or following any scientific method to speak of. They use smoke and mirrors, misdirection, and misrepresentation of “findings” to tell the story they want believed instead of informing and educating the readers. I’d like to think that individuals who perpetuate this type of pseudoscience started with a genuine concern for wellness and wellbeing, be it human or animals, but have got themselves so entrenched in their beliefs that they lost sight of reality.  So how can you as the concerned pet caregiver tell fact from fiction? One way is to read everything with a healthy dose of skepticism.

In the world of true scientific publishing, the peer-review process helps act as a critical eye before studies are printed. This process is not perfect and occasionally bad or biased studies are released, but overall the peer-review process is an important step in ensuring the integrity of published science. When studies are submitted to a peer-reviewed journal, the entire materials and methods used by the researchers are outlined in detail. This allows the reviewers to examine and critique the study looking for bias in methodology or omission of important details, and whether the researchers conclusions are supported by the finding. Some journals have started requesting the raw data so that the reviewer can look for irregularities themselves or even repeat statistical analyses. If these reviewers miss key details and a paper is published with gaps in methodology or with conclusions not supported by the data, the journal’s readers can write to Letters to the Editor outlining their critique of the article in question. The researchers then have a chance to respond or even reconsider their stance. Sometimes published research papers are revised and re-published and sometimes fatal flaws are reveal and the paper is removed from reference libraries (just like the retracted and discredited “study” linking vaccines to autism).

“Scientists will hold up their research for critical evaluation while Hucksters will hold up the fact that something is printed as evidence of it being true.”

If a company or individual prints their study on a website, or in conference proceedings (meaning that the researcher or a representative of the researcher talked about their finding at a meeting somewhere), or in marketing materials, the critical review stage has been by-passed completely. The messy details of methodology and actual data are typically omitted and we as the reader have no knowledge of how the study was conducted or whether there are critical flaws that would invalidate the results. The reader is expected to accept the researcher’s conclusions as truth without the opportunity to review the methods or data, or even evaluate whether there are conflicts of interest between the researchers and those “publishing” the study. Scientists will hold up their research for critical evaluation while Hucksters will hold up the fact that something is printed as evidence of it being true. In my opinion true scientists are willing to admit the fact they don’t have all of the answers, which is why they keep asking questions.

Testing the hypothesis…

What do I look for in a published research study? I want to know where the tests were performed, how they were performed, how many samples were used, what they used as positive or negative control (controls are when help ensure that environmental contamination is not occurring which can bias the findings), and what statistics were used (statistics help differentiate random chance from true relationships). A well written Material and Methods section will allow another researcher to replicating the study. This is how the field of science grows, by testing hypotheses and challenging assumptions. The absence of Materials and Methods and any statistical evaluation in any study (whether submitted for peer-reviewed or “published” online) are bright red, waving flags for me.

Not that pet food manufacturers or owners should be blasé about potential problems in pet foods, but without any study details we have no way of knowing whether the researchers grabbed one dented can on the shelf, did one fungal test and then stopped testing when they found a (weak) positive result, or whether they tested multiple times or different batches of foods in each brand and repeatedly found the same result. Even if you were to trust their laboratory procedures on blind-faith, the more detailed section on bacterial contamination admits that the bacteria found were the result of improperly refrigerated food. Yet in the infographic, the reader is presented with a table outlining how almost all of the products tested positive for bacteria. Could that also be true for the fungal toxins? Everyone (hopefully) knows that if you leave any food exposed to room air at room temperature it will start to grow things, but improperly stored and handled foods by the consumer is very different than poor manufacturing practices.

“improperly stored and handled foods by the consumer is very different than poor manufacturing practices”

Let’s look more closely at the reported risk from fungal contamination. Mycotoxins (toxic compounds from fungi growing in and on foods) are a serious problem in foods and ingestion of even small amounts per pound of body weight can causing serious health effects and even death in people, dogs, and cats.  Again, if we accept the tests were done correctly and the results were not a factor of improper storage and handling, when you read the details about the fungal finding it is only when you add all of their ‘low’ risk numbers together that the ‘risk values’ get worse. I haven’t taken a statistics course in almost 20 years, but I’m pretty sure you can’t just add numbers together to provide significance.

As for the ‘nutrient’ concerns, let me start by saying that dogs and cats do not have sulfur requirements and there is no lower or upper limited in the 2006 NRC or any version of AAFCO for sulfur because it is not an essential dietary nutrient. There are requirements for the sulfur-containing amino acids (methionine, cystine, and taurine) and sulfur is found in connective tissue, which is a part of any meat, so it is not surprising that it was found in the diet. (I would be more surprised if it wasn’t.) Excess calcium intake on the other hand is a problem in growing large breed dogs, but since the foods that tested higher in calcium were either cat foods or adult dog maintenance diets, neither of which should be fed to a puppy, I’m also not worried about these nutrient values.

This “study” also compared nutrients in foods on a Dry Matter basis without providing Calorie (kcal) contents of the foods tested, but direct Dry Matter basis comparisons between foods are problematic and not used routinely by small animal (or human) nutritionists. Dogs and cats (and people) will eat to meet a particular energy requirement each day while the mineral requirements are independent of energy intake. The calcium content of an energy dense diet should be higher than the calcium content of a low energy diets to ensure that the animal received enough calcium per day despite eating less Dry Matter. To put in simple terms, if a dog needs to eat 3 cups a day of Brand X dry food to meet energy needs, but only 2 cups of Brand Y dog food to get the same Calories, then Brand Y must add 50% more calcium per cup to ensure that the dog is receiving the correct calcium intake per day despite having a lower Dry Matter intake. Providing Dry Matter values alone is meaningless.

Veterinarians and caregivers need to be advocates for their furry family-members and access to accurate, unbiased information is an important part of that. So here is what I consider my Pet Food and Supplement Red Flags:

# 1: Results promoted through media and not scientific journals.

#2: Miracle “cures” that are not supported by the larger medical or veterinary fields.

#3: Proprietary or secret formulas or methods.

#4: “Proof” that is provided by testimonials instead of facts.

Education is key, but so is checking the references

Happy Feeding!

Lisa P. Weeth, DVM, MRCVS, DACVN