In Part 4 of Reading Between the Drops we look at the journals that publish research papers. And we discuss studies that look at metadata.
Journals
Scholarly journals publish research papers. Sometimes the journals are peer-reviewed. Peer-reviewed means the paper was judged and approved by a jury of experts before publication. The review process gives credibility to the paper.
It’s not always obvious which journals are, and which journals aren’t, peer-reviewed. You can check the journal’s masthead, the section with the names of the editors. But you might have to dig deeper to find out for sure. Usually peer-reviewed journals display their status prominently. So if you’re searching, and not finding any evidence of peer-reviewing, it’s probably not peer-reviewed.
(If you have access to it, Ulrich’s (ulrichsweb.serialsolutions.com) let’s you check if a journal is peer-reviewed).
No Standard Peer-Review Process
Even so, not all peer-reviewed journals are created equal. That’s because there’s no standard process all journals follow to ensure the quality of their reviews. So one journal might have a rigorous peer-review process. But another one’s might be more lax.
How so?
It might start with something as simple as independence.
Independence means different people perform different roles. Authors, for example, can’t be editors or reviewers.
But not all journals require independence. That means you could be the author and the reviewer of your own paper.
Can you imagine what could happen if you were judging your own paper. Would you challenge your hypothesis? Would you challenge your methodology or your conclusions?
In two words: probably not.
Or more accurately, definitely not.
As a result, you’d be able to publish whatever you want. There’d be no one to challenge your assumptions or the design of your study. No one would question your conclusions, or anything else about what you’re saying.
Reading Between the Drops – 7 Questions to Ask
1. Does the hypothesis make sense?
2. How the research was conducted, does that make sense?
3. Does the conclusion make sense
4. Who are the authors?
5. Did the authors make any financial disclosures?
6. Is the journal peer-reviewed?
7. Is the journal independently peer-reviewed?
And sometimes a lack of independence means important studies don’t get published. This sometimes happens when editorial boards have their own agendas. If a study comes along that challenges these agendas, the study might end up collecting dust. Lots of it.
To be sure, sometimes a lack of independence happens for a very good reason or it’s unavoidable. There just might not be enough experts in the field to go around, for example. Something like that might happen if it’s a new field or if there are only handful of experts. Even so, it could indicate a problem.
Analyzing Metadata
Metadata. Studies about data in other studies. Metadata studies look at all of the available data on a particular topic. They aggregate, filter, and make sense of the data. Simple enough, unless it’s not.
Because, as noted above, there’s always the potential for bias. Editors might block some papers or make publishing all but impossible. Unfortunately, it happens all the time.
So by definition, a meta analysis can only consider published, not suppressed, data.
And that means just one previously suppressed study could throw a metadata study out the window.
Wow! That could change things… a lot!
Just think about it.
And with that thought, we conclude our 4-part series on Dry Eye studies.
Send your questions and comments to blogger@notadryeye.org.
Subscribe to our blog to keep up with Not A Dry Eye. It’s FREE!