At times, the medical literature can seem a bit like North Korea. Everyone is on the same team, wearing the same outfit, sporting an identical hairdo. The articles are written by doctors who believe in the basic doctor approach to everything: more health care, fewer vitamins. The readers are doctors who also believe in the same basics or else student doctors waiting to be branded. Though there are no explicit photos of Our Dear Leader or his medical equivalent scattered through the pages of JAMA or The New England Journal, it is difficult to ignore the dominance of the monolithic, unelected chorus reinforcing the basic soundtrack.

So who can be surprised by a recent medical article that sets out to demonstrate that Wikipedia, the omnipresent brain of modernity, is not as good a doctor as a doctor is? Turns out that the Wiki entries, written by those invisible people who know everything, have some mistakes, or at least have, in their medical entries, what the authors call “assertions” in variance with what the best evidence suggests should be done.

Writing in The Journal of the American Osteopathic Association, investigators selected the Wiki entries on 10 common and costly conditions, including back pain, diabetes, and coronary artery disease. They extracted the guts of the assertions for each condition and then ran the medical literature to see what evidence-based guidelines and articles had said. Unlike previous articles that examined the Wiki-medical nexus and found solid concordance, these investigators found that, for nine of the 10 indicator conditions, the assertions found in Wikipedia were not fully supported by the best evidence.

The article is pretty light on the details for such a provocative headline. Their methods raise a few eyebrows, too. For reference, they directed their reviewers to an online textbook called UpToDate, a multi-authored e-text with big shots and moderately sized shots writing the entries. (Full disclosure: The UpToDate people never have asked me to contribute, though a friend did ask me to write a section with her; I declined, fearing the demands of endless up-to-dating.) Using UpToDate as the ur-text, though, gets into a vast expanse of gray zone, with some assertions evidence-based but many not.

Indeed, the Cochrane Collaboration, an enterprise in the business of sorting through medical information to find useful conclusions, estimates that most medical decisions are not supported by the evidence. And not for not trying—simply not enough evidence is out there. Randomized clinical trials needed to clarify areas of confusion are slow, costly, and uncertain. Remedies longa, vita brevis. So in many cases of discordance, it may be a Wiki author’s view versus the UpToDate, view with neither standing on firm ground.

There are several other things in the Wiki vs. doctors article to quibble about, which the authors generously note. They used physicians in training to determine what fit the literature and what did not. They didn’t use enough reviewers for the Wiki articles to really filet them into component pieces. But these are routine limitations to studies, all of which could be done better but for the multiple moving parts of medical care.

The large problem, though, the one that makes this and similar studies fun to talk about but useless to rely on, is that the authors are not comparing the Wiki-based assertions to what a doctor does minute to minute. These investigators and others on the topic set a practicing doctor as the gold standard of correctness—as if, like the customer, the doctor is always right.

Um, well, gee, guys, thanks for the vote of confidence, but neither opinion nor evidence suggests that doctors themselves always work from an evidence base. There is medical literature a mile long detailing our deviations and dismissals of guidelines and nationally endorsed approaches. We might not be off the mark nine out of 10 times, but we deviate plenty.

Which is not to say we are fat slobs. We doctors do our best, honest. We try to make decisions that are clear, thoughtful, and evidenced-based but, as George W. Bush once said about being president, it’s hard work. Doubt and panic and regret haunt the heart of any decent physician who ever has made a decision.

So the real comparison is not between Wiki and the latest calmly published guideline vetted by experts, smoothly uploaded onto a website, full of references and figures and interactive media. It’s between Wiki (or a guideline or UpToDate) and a doctor who perhaps made a mistake yesterday. He isn’t sure but is creeped out by the possibility, is worrying about how a different patient is doing in the hospital just now, and has a patient coming later to see him whom he never has liked but now, this moment, has to make a decision about giving an antibiotic to this coughing patient who had a fever last week but not now and is too busy to go for an X-ray and has old antibiotics at home and can’t he just take them?

Wiki and guidelines and 1,001 medical articles all fall short of fully informing most health-related decisions for the same reason: None can account for center of the problem, the brilliant chaos of human beings, patient and the doctor alike.