Time Scavengers

Kristina here –

Today, humans have access to more information than at any other time in human history, all at the tips of our fingers with a quick Google search, or a “Hey, [insert AI name here]”. While equal access to the internet and information technology is beyond the scope of this post, cell phones, tablets, and laptops, have made it easier than ever to quickly look up information. Yet with this technology has come a huge surge in wide-spread misinformation, making it difficult to know whether you can trust the information you find. Pretty much anyone can post whatever they want, and pass it off as “fact”. How then can the average person determine whether what they’re reading is actually credible and factual? Furthermore, if you see something that says “scientists disagree on X topic”, who should you believe? Contrary to what you might think, not all viewpoints are created equal, and both scientists and the average person can be guilty of confusing “opinion” and “fact”. This is where the “peer-review” process comes in to help.

So what is “peer-reviewing”?

Most people hearing “peer-review” assume it is a good thing (and this is certainly true) but what does “peer-review” mean? Essentially, peer-review is an integral part of the scientific process, and what helps separate “opinion” and “fact”. It is what scientists use to make sure that their research is as thorough, accurate, and factual as possible. In general, scientists do not consider something trustworthy or credible unless it has gone through some kind of peer-review process.

How does peer-review work?

A scientist or group of scientists will first go about conducting research. They will ideally do background reading to make sure they understand what is already known about the topic, and where there might be gaps in our knowledge. They will then design an experiment or test, collect data, and analyse that data. The ultimate goal of science is to try and refute a null hypothesis (e.g., all apples are red). We must prove beyond a reasonable doubt that something is different from the null (what has been previously determined) (e.g., some apples are green). If we can’t prove otherwise, and/or the more scientists that run their own tests and come to the same conclusion, the stronger our hypothesis is, or the closer it is to “the truth” (e.g., apples can be different colours).

Once scientists are finished collecting and analysing the data, and have come to a conclusion (e.g., refuted, or failed to refute a hypothesis), they will write a paper reporting their findings. See Sarah’s post on how to write a scientific paper here. The authors then submit the paper to a peer-reviewed journal, usually one that has been selected based on the topic or audience of the journal. The submitted paper is sent to an editor at that journal, who then decides if the paper is appropriate for their journal. If the paper “passes” this first test, the editor will then send it out to at least two experts in that topic.

How are the peer-reviewers selected?

Usually, journals request that authors include anywhere from 2 – 10 names of experts that know enough about the topic to provide sufficiently thorough critiques of the paper. Authors cannot include close colleagues or collaborators, as this could create bias (e.g., your friend is more likely to give you a pass, even if you don’t deserve it). Editors can opt to choose as many or as few people as they want from the authors’ list. Ideally, editors will also find at least one person not on the authors’ list that is an expert on the topic. Authors may also include a list of people they don’t want to review their papers, but they must have a good reason (e.g., “this person doesn’t agree with me” is not an acceptable reason as critical reviews are important to ensure scientific rigor. But, “this person has been openly hostile towards me” would be – some people can be jerks and block good science in peer-review). If you have too many people that you don’t want to review your paper, that sends up red flags to editors, however, so including people on a “no-review” list shouldn’t be taken lightly by authors, and should only be done when absolutely necessary.

The editor then sends the paper to at least two of these reviewers. If the reviewers accept, they then have about 2 – 4 weeks to evaluate the paper. It’s important to note that editors do not usually review the papers themselves (unless they happen to be an expert in that topic), because, especially for larger journals, the editor is unlikely to know enough about the topic to give sufficiently thorough feedback (e.g., a vertebrate palaeontologist won’t review an invertebrate palaeontology paper, and vice versa). 

Peer-reviewing a scientific paper

If you are the reviewer, your job is to go through the paper and evaluate the science independently. Your comments should stick to the science and presentation of the science, and you must refrain from unnecessary criticism of the authors. For example, “this is a poorly written paragraph” is not helpful or appropriate. Instead, you should point out where you didn’t understand what was written, and why. Reviewers typically read the paper over several times to make sure they understand what the authors are trying to test, then evaluate whether the experimental design, methods, and analyses of the data were sufficient to test the hypothesis. Often, reviewers will analyse the data themselves to make sure they find the same things as the authors. Sometimes, if the reviewer feels that the methods or analyses were insufficient, they will suggest that the authors try other analyses that will more accurately test their hypothesis. This is one of the most common types of reviewer feedback. 

If the methods and analyses all hold up to scrutiny, the reviewer will then make sure that the interpretation of the data (included in a paper’s “Discussion” and “Conclusions” sections) matches the results of the analyses. Another common type of feedback from reviewers occurs when authors overstate (or sometimes understate) their conclusions (e.g., the authors may claim their paper proves x, but their results might only be applicable under very specific circumstances). A good reviewer will make sure that all of the claims made by the authors are supported by the tests they perform, and should watch for speculation (speculation may be acceptable, so long as it is clearly stated that it is such).

Reviewers then provide a thorough report back to the editor, including specific comments/suggested edits from throughout the paper. Reviewers will provide a recommendation to the editor indicating whether they think the paper is in need of revisions (“major” or “minor” revisions), or if the paper should be rejected or accepted. Major and minor revisions are the most common reviewer recommendations – major often means further analyses are needed before the hypotheses have been sufficiently tested, minor usually means that the methods and results are sound, but the authors need to tweak a few paragraphs, interpretations, or graphs throughout the paper. Papers that are considered “accepted” are exceptionally well done, and the reviewer may only have small comments that need to be addressed, or possibly none. Papers that reviewers “reject” usually have insufficient evidence to accurately test the hypotheses proposed, may have critically flawed methods or analyses, or would require very extensive revisions that would take a long time to complete, or would end up testing a different hypothesis. Rejections do not always mean that the authors should abandon the paper – it could just mean that there is more work to do before the paper can be fully evaluated. Some journals even have a “reject and resubmit” option, which means that the paper is rejected for now, but that the authors are welcome to resubmit in the future if they are able to address the reviewer’s concerns. It is sort of like “very major revisions” and gives the authors a bit more time/flexibility to complete the revisions.

Revisions

Once the editor has received reviews back from all of the reviewers, they will go through all of them to see if the reviewers have picked out common flaws in the paper, and to make sure the reviews were sufficient. If the reviewers clearly disagreed on something, the editor will often send out the paper to at least one other reviewer for another opinion (this is helpful if a reviewer was unnecessarily harsh or lax). Based on all of the reviewer evaluations, the editor will provide the final recommendation for the paper (accept, reject, reject and resubmit, major/minor revisions). The editor then sends their recommendation and summary of the reviews, along with all of the reviewer comments, back to the authors. 

The authors must then revise the paper based on the reviewer feedback, and address every single comment made by the reviewers. It is the job of the authors to not be defensive about the comments (which can be hard when someone is criticizing your work), but it is important to remember that the reviewer’s job is to make your science better. Depending on the amount of revisions requested (major or minor), the authors are usually given at least 2 weeks (and sometimes several months) to provide their revisions, as addressing every single comment thoroughly takes time. The authors then resubmit their revised paper, as well as a list of their responses to all of the reviewer comments and the actions taken to address each comment. The editor uses both documents to determine if the authors have done due diligence with the reviewer’s feedback, or if further revisions are needed.

If necessary, the editor will send the paper back to the original reviewers or to new reviewers. The process will repeat until the paper becomes acceptable to reviewers, or the paper is rejected. Once the paper is considered acceptable by the reviewers and editor, the peer-review process is complete and the paper is ready to be formatted and published in the journal! It can take anywhere from weeks to years for a paper to become accepted! 

Responsibilities of reviewers

It is important to note that neither authors, nor reviewers are paid (editors at larger journals are sometimes paid positions). Instead, peer-reviewing is considered an “academic service” and authors should expect to review 1 – 2 papers for every paper they publish (i.e., for each review of your paper, you should return the favour by reviewing that many papers). While some people have strong opinions on monetary compensation for reviewers and editors, the current justification is that reviewing is a service and the lack of compensation should keep reviewers impartial. The peer-review process is a lot of work for everyone involved, but is the best way to ensure we have a system that produces sound, thorough, and accurate science.

Peer-review doesn’t just happen in journals, either. Scientific books, text books, theses, and government reports may also be considered peer-reviewed, as they are usually thoroughly reviewed by several experts, or scientific review panels. But the most common form, and the most acceptable form of citations or sources, are peer-reviewed journal articles. Peer-review also occurs for articles in other fields in academia, such as history and the arts.

So, how do you tell if what you’re reading (or what you’ve heard) is credible?

Has it been peer-reviewed?

Is the information coming from a reputable peer-reviewed journal?

Do they cite their sources when stating information/presenting facts?

Even if the information isn’t presented in a journal (e.g., a governmental report, book, or blog post), do they use citations to support their arguments? Are these sources credible (i.e., from peer-review sources, not some random internet link)?

Do the majority of scientists/experts in the topic agree with this opinion?

If you come across a “fact” that a scientist has stated, remember that not all “opinions” are created equal. If the majority of experts have come to a conclusion, yet one person disagrees, that person has most likely failed to properly refute a hypothesis (their conclusions do not match the majority of the evidence). This usually happens when a scientist fails to include all of the appropriate variables in their methods, meaning that the test they used to refute the hypothesis was flawed, even if their work has been published. For example, those that claim global warming has happened before and that therefore the global warming we are experiencing today is just natural variation are failing to include an important variable: the rate of change (which is much faster than any past “background” variation). 

Is the author an expert on the subject?

It takes several years to gain expertise in a topic, mostly by reading all of the peer-reviewed papers on that topic (hundreds or even thousands of papers), staying up-to-date on new research, conducting experiments, and going through the peer-review process. Google searches don’t cut it. Even if they are a scientist, if they normally work in a different topic, there is a greater chance that they might be missing something that is common knowledge to experts in that field. For example, I as a palaeontologist am not about to try and write a paper on black holes, even though I think they’re fascinating and have read lots about them. 

An imperfect system

The peer-review system is not infallible. Nowadays, scientists will often “publish” their work online outside or ahead of the peer-review process with things called pre-prints. Pre-prints allow scientists to share their work, especially large datasets, ahead of peer-review so that they can share their work more quickly and potentially get feedback from other researchers. Often, the data included in pre-prints will end up going through the peer-review process, but as the peer-review process can take a long time, pre-prints allow researchers to get their data out there and get feedback faster. While it may not seem as rigorous because it hasn’t gone through the peer-review process, it can actually end up being more transparent because it potentially allows more people to review the research. Essentially, pre-prints still go through “peer-review” in the actual sense of the word, just not necessarily through the traditional channels of journals.

Journal reviewers can also sometimes act inappropriately. For example, reviewers might make unhelpful comments that are not constructive or based on the science, or may even be downright abusive or derogatory – e.g., criticizing the author, not the work, or saying something unnecessarily rude. While these kinds of comments are not permissible in the peer-review process, and it is usually the responsibility of the editor to reject reviews that include inappropriate content, these kinds of things regularly slip through. It is then within the author’s right to ask the editor to step in and find an alternative reviewer or to ignore the comments when making their final decision. These kinds of checks and balances are what help the peer-review process to remain as impartial as possible – comments must be limited to the science and the presentation of material, and cannot include opinions or feelings about the work, even if it disagrees with your own.

Finally, just because something gets published doesn’t mean it’s perfect. There are lots of bad papers out there that slip through the peer-review process. Editors and reviewers are people too. That is why scientists must always evaluate previous work for themselves. It is an inherent part of the scientific process – trying to independently reject that null hypothesis to see if you come to the same conclusion.