Suggestions for critical paper/presentation review

[annapurna]

There seemed to me to be an increasing desire to dig up and quote studies to support or refute various positions on the board and to do so without a full understanding of the technical merit of those papers. While neither Laura nor I are doctors or any form of medical professional, a large fraction of our daily job is sifting through an enormous amount of data on various problems to do with Space Shuttle work and picking the pertainent information. This daily effort has given us far too much experience with removing the useless BS from someone's argument and picking out the (far too often) too few nuggets of truth. With that in mind, I'd suggest the following guidelines for questions to answer before you decide how much to trust a paper.

1) Who paid for the research and expense of publishing the paper? Often finding out who paid for the paper can tell you whether or not you should expect biased research or incomplete and slanted analysis. I'm not indicting specific manufacturers with this. Everyone pays for and publishes papers that benefit them. Some go to an extreme and accept poor techniques and poor analysis to prove a point, others simply do research as pure and untainted as possible but only publish the papers that defend their positions.

2) The same question should be asked about who did the work. There are some cases where the researcher isn't directly connected with the payer. It's rare but it does happen. Understand the researcher's bent as well as the payer's.

3) Where was the paper presented? Some places where the paper might be presented are carefully reviewed by other researchers for technical content, some receive a cursory review, and some papers are published without review. Some conferences permit longer (20 minute) talks with questions, other permit short (5 minutes) with limited questions. Ideally, the best would be a peer-reviewed paper presented with extensive questions from the audience, but even then basic mistakes might not be caught. I say this with assurance as I've seen papers in my field get through the system with logic errors and poor testing technique.

4) A second part of the previous question is whether or not you are reading the entire paper, an abstract, an extensive but summarized presentation, or a short presentation. Obviously, all of the details supporting the conclusions tend to be the first thing cut out in summarizing the work.

5) To begin analyzing the actual work done, the first question is to ask what the original question being answered was intended to be. If the researcher is lucky, he/she managed to answer the question in its original form. Typically, the data gained didn't quite manage to answer the original question completely and just the supported conclusions were answered. If you read through the paper, try to get a feel for what the original intent was. That gives you an idea of what kind of data were gathered and what kinds were probably omitted from the research. Really, the thing to look for is a study that was biased from the start, intentionally or unintentionally, because the researcher didn't gather the right kinds of data to really answer the question he/she had in mind.

6) Similarly, you should understand what the actual work done was. Did the research gather data from other papers and present a summary article? Was it a clinical study? Was it a lab-based test? If the researcher did a lab-based test or clinical trial, does the work done make sense to you as the kind of work you'd do to answer the question they had in mind? Did the researcher producing a summary article include all of the papers on a subject or did he/she exclude certain kinds of papers? Does that bias taint the conclusion they came to?

7) Abuse of statistical sample size is rampant. Basically, this just needs to pass the gut-feel test. Did the researcher test enough cases to compare with the entire body of potential people who would fall in that category? A test of 10 ADR patients doesn't work when you wish to make a generalized statement about all ADR patients. A test of a single patient is sufficient to say that an observed condition CAN happen but it takes lots of people to say how LIKELY something may be.

8) Statistical analysis of the data is often abused to justify conclusions. A detailed discussion of what to look for is beyond my knowledge but, in general, better papers should have a discussion of what kind of statistical treatment was used to reach a conclusion. When the test involves real people, you can expect so much variability from person to person, the conclusion should either be so blindingly obvious from the data or supported by so many people tested that arguments aren't likely. If you see small sample sizes (few people) or complex conclusions, be skeptical.

[Harrison]

All excellent points. Thanks for sharing your ideas, all of which are interesting. I’ll add my comments shortly.

[Justin]

Points 1-8 above are well thought out and bring many important issues concerning publications/research to the table. FYI the articles I post or post links to are from peer-reviewed journals. To have a publication in Spine, for example, is no easy feat. Most of the papers submitted for review hit the trash before anyone takes a serious look at the publication. I worked with a physician that publishes a lot of papers--his fulltime job is to write papers and grant proposals. This takes so much time and effort that he only sees patients once a week.

He explained to me how most of his papers don't reach publication, but the ones that do are worth all the effort. (Trust me this guy is brilliant too--he has won multiple awards for investigator of the year, etc.)

I'll chime back in later before I get too off topic...I gotta run.

Justin

[annapurna]

Peer review helps quite a bit, but I'd suggest that it doesn't resolve all of the problems. I've personally sat through junk in technical conferences in my field in what was supposed to have been a reviewed article. My best guess was that they couldn't find anyone competent to review those articles because they were all busy doing real work and the one I saw were reviewed by what was left. The bottom line I'd propose is that the paper needs to make sense to you for you to be willing to trust it. You might not follow all the details but it feels like BS, then it probably is.

[Harrison]

It will be interesting to see how the newer incarnation of peer review will help disseminate the information AND research AND conclusions.

For example, in a very short amount of time, wikipedia has become a global reference source. This success may explain, in part, the rollout of "open peer review." More on this here:

http://www.healthpointcapital.com/research/2006/10/04/a...oach_to_peer_review/

[Justin]

The scary part about Wikipedia is that it is not peer-reviewed (open peer review). ANYONE can alter the content of the material that is posted. Don't get me wrong, wiki is a good idea, but it is not a resource that should be viewed as completely accurate. However, many journals are utilizing review online--but this is done by scholars (or those who are qualified to review such publications) and not laymen.

A brief list of "basic" things to look for in publications as a patient:

**P Values less than 0.05 (represents statistically significant findings)

**Clinically relevant (number needed to treat, number needed to harm, absolute risk reduction, etc.)

**If the study is double-blinded, randomized, etc.

**Sample size (population)

**Adverse events / complications

Justin

[ans]

My bro, a physician, told me that I'll have trouble reading in b/w the lines of medical journals. This is an older article about conflict of interest and I wonder how things are now post-the NIH publishing scandals.

http://www.tufts.edu/~skrimsky/PDF/conflict.PDF