Like many things in the world of cannabis, the research landscape has changed a lot over recent years. Despite lingering difficulties due to prohibition, more and more cannabis research is being done every day, both in the US and globally. That’s great news — and not just for cannabis researchers, consumers, patients, advocates, and aficionados. These discoveries are valuable to us all.
The results of new scientific studies in cannabis are often headline makers. We’ve all read “according to a new study” or “research shows,” but how often do we go back and look at the actual study? Research papers can be dense and require a different approach than reading articles or blog posts about said research, but what you find by going to the source might surprise you.
Here’s how to fully scrutinize and read cannabis research studies so you can confidently go straight to the source.
This is perhaps the most important step of the whole process when reading scientific studies, since looking at who did the research, why they did it, how it was funded, and who published it can provide some clues about the motivations of the researchers and overall quality of the research.
Aside from identifying which journal or publication published the article, you can skip to the end to see the authors’ affiliations, how the research was funded, and any conflicts of interest.
Peer-reviewed journals
“One of the first things I do is look for peer-reviewed journals,” shared Dr. Chris Hudalla, an analytical chemist and Chief Science Officer at cannabis testing lab ProVerde Laboratories.
Most often, when a researcher conducts a piece of serious research — whether it be a trial, a study, or a survey — they will publish their findings as a paper in a peer-reviewed journal. These journals, like Nature and The New England Journal of Medicine, for example, are publications with a reputation for bringing valuable and valid information to the scientific community and public.
The peer-review process enlists independent experts in the field to assess submitted manuscripts for validity and significance and to help editors determine whether a manuscript should be published in their journal. The reviewers can offer suggestions for refining the analysis or claims, or they can reject the manuscript entirely.
“It’s a process that has worked, but there are also peer-reviewed articles that have looked at the peer-review process and found that it isn’t always perfect,” added Dr. Dustin Sulak, an integrative medicine physician based in Maine who has been using cannabis in his practice since 2009.
It’s true, the peer review process itself isn’t perfect: sometimes bad science slips through the cracks. The value of peer review is an ongoing debate in the scientific community, and some would argue that there are very real problems with it. The rigor that research articles go through varies from journal to journal, and there are now tools that can help augment the peer review process by eliciting feedback on research before and after peer-reviewed publication.
For the most part, though, many scientists — and all the sources I spoke to — agree that checking if a paper has been peer-reviewed is a good first step in weeding out good research from the bad.
Preprint articles
Researchers can also post their manuscripts online before formally submitting them to journals. These “preprint” articles are just that, they have not been vetted by any form of peer review and have not yet been accepted by a journal.
For most authors, the preprint is not the final research paper. You can think of it as a draft. The value, for authors, of posting preprint papers is to elicit feedback from their peers before submitting to a journal.
Impact metrics
There are other metrics you can use to get a sense of a journal’s reach and reputability, and the two biggies include the Impact Factor and CiteScore. These metrics measure the number of citations a journal has had over a period of time. Keep in mind that these metrics are just one piece of information, some quality publications might not have such a score, and sometimes predatory companies make up fake versions of these metrics.
Predatory journals
Predatory journals, also referred to as questionable journals, is a catchall name that describes a variety of different publications, but the through-line is that they are essentially bad actors: scamming the researchers seeking publication and/or scamming the public.
These publications will take publication fees from authors and then provide little to no peer review process or editorial service, and to the unknowing public the research passes off as credible simply because it’s published.
There are many different types of research and many different types of research articles.
When it comes to cannabis research, folks will often come across or seek out original primary research articles. Most of these papers have a similar basic structure, and will consist of a few standard sections:
- Abstract: A brief and succinct overview of the entire study.
- Introduction: Provides context for the study with a summary of relevant prior research, and the unresolved issues that the study will address.
- Methods: A detailed report of how the research was performed, including details on the study’s design, the participants or subjects involved, the materials used, and the procedure.
- Results: A detailed report of the data that was collected and the results of the data analysis; it will probably include charts, tables, graphs, or other presentations of the key findings.
- Discussion: Where the researchers discuss their major findings, how those results relate to the big question they aimed to answer, and expand upon the implications of their findings; they should also address limitations of the study or its design.
- Conclusion: A conclusion will pull together the entire paper by reviewing the major findings. It will often discuss how this research adds to the existing body of literature and directions for future research.
- Author & Article Information: Here is where the authors will list their affiliations, how and by whom the research was funded, and any conflicts of interest.
- References: A list of any research, books, or other sources cited.
- Additional supplements: Any additional details, graphs, figures, or data that will help with a better understanding the research but not critical to include in the “Results” section.
Digging into a research article can be daunting. While it might be tempting to read the easy and succinct abstract and call it a day, the abstract does not make a fully informed opinion. Though, sometimes, an abstract is all a researcher will publish to make a position or inform the community about what they are doing.
Once you’ve looked over a paper and decided it seems reputable and robust, all of the scientists I talked to recommended reading the paper out of order. Each of them had their own plan of attack, so there is no wrong way to go about it.
“Usually what I tell non-scientists is to read the title, the abstract, and then the discussion and conclusion,” suggested Dr. Jahan Marcu, cannabis researcher and Editor in Chief at the American Journal of Endocannabinoid Medicine. “That will allow you to familiarize yourself with the most accessible language in the paper. If you need more information or want to dig deeper and really understand how they found the conclusions, then you can get into the methods.”
Dr. Jennifer Raff, geneticist and Associate Professor at the University of Kansas, wrote a detailed and robust guide for reading research papers for the London School of Economics blog, and offers a condensed version of this guide as well. She recommends skipping the abstract until the end since it can inadvertently bias your interpretation of the results.
Both Raff and Sulak agree the introduction is a good place to start for context. Here, you can learn more about the authors’ ethos and motivations, along with getting a sense as to the existing research, which is usually referenced in the intro. This can help you pinpoint other important studies in the field, which you might want to look up, too.
Dr. Raff suggests identifying the big question first: what is this entire field trying to understand? Then, identify the specific questions this paper sets out to investigate.
After the introduction, Dr. Sulak likes to look at the study’s limitations, usually covered in the Discussion portion, where the researchers discuss the flaws or shortcomings related to the study’s design or methodology. He follows that with information on the authors’ affiliations and the study’s funding — usually found at the end of the paper — before diving into the rest.
If you want to really get your head around an article, Dr. Marcu personally uses a three-pass strategy. “Read it all to get a general sense of it. Read it again, stopping and underlining the things you don’t understand, and then look all that stuff up, then go back and read the whole thing again — then you will finally really understand the paper.”
The Methods and Results sections can be the most challenging to work through, even for scientists working in the field.
“When I am consulting or working with patients I often tell them to ignore [those parts] and instead focus on the conclusions,” said Dr. Hudalla. “Especially if it’s a respected author and institution who is published in a peer-reviewed journal, looking at the confidence intervals, p-values, or correlation coefficients is enough to make your eyes glaze over.”
That said, here are some considerations for tackling the dense parts of a research paper.
Methods
The method explains how the study was designed and carried out. This section should be detailed enough for another researcher to replicate the research.
Depending on the type of research, any number of methods could be used. If you’re looking at medically focused cannabis research, you might see terms like “in vitro,” “in vivo,” and “clinical” referencing the type of study that was done.
- In vitro studies are done outside a living organism, for example looking at cancer cells in a petri dish or the sensitivity of bacteria to antibiotics.
- In vivo studies are done on living organisms, and often refer to animal research.
- Clinical studies or trials are done on humans.
Each of these is valuable and useful with their own pros and cons. The flow of research often goes from in vitro, to animal studies, and then to human trials. Sometimes, what’s observed in vitro doesn’t translate to in vivo.
“Each phase of research also gets exponentially more expensive and time-consuming, which is why new drugs take forever to hit the market and cost pharmas a fortune to develop,” added Dr. Hudalla.
You might read about specific animal models that were used in the methodology, and many animal models that have been developed and established as reliable for mimicking human conditions or diseases like cancer, pain, autoimmune disorders, and epilepsy, for example.
When it comes to human clinical trials for drug development, the “gold standard” is the randomized controlled trial (or RCT). These studies are designed to randomly assign and compare two groups, one receiving the treatment being investigated and another receiving a placebo. Running these trials is extremely resource intense, both in time and money.
Observational studies, compared to RCTs, observe individuals without manipulation or intervention from the researcher. Surveys or longitudinal studies are kinds of observational studies you might come across in cannabis research.
Results
The Results section can include the most challenging pieces of the paper, particularly when they get into detailed descriptions of the data analysis and statistical techniques used.
This is where all of the scientists I spoke to agreed that if you trust the authors, institutions, and journals publishing the research, you don’t have to get too hung up on understanding it all in great detail. The discussion and conclusions will have distilled the stats and analysis into a few concise paragraphs.
You will almost always see the terms “significant” and “non-significant” and these are referring to the results of statistical analysis. When a result is statistically significant — or you see a p-value equal to or less than 0.05 — it means that there is a “95%+ chance that we are not kidding ourselves,” said Dr. Marcu. In other words, there is a high probability that the results are not due to chance.
Another thing to keep in mind is the sample size. There might be statistically significant results, but the sample size was ten people and made up of only healthy young males. For some studies, this might be sufficient, but for most, a bigger and broader sample is better.
Dr. Raff offers questions to frame the results section: Do the results answer the specific questions they sought out to investigate? What do the authors think the results mean? Do you agree?
Don’t understand a specific word? Look it up. Is there a cell or animal model, a cognitive test, or a depression rating scale mentioned that you have never heard of? Look it up.
The more you do, the more you will start to recognize common methodological tools used in research — especially if there is a particular area of cannabis research you’re interested in learning more about.
While the introduction might have given you a snapshot of the research landscape related to the paper or topic being studied, the references list will give you a bigger picture view. Even just scanning the titles of other papers in the list will give you a sense as to what researchers are focused on in this field, and how this paper fits into that landscape. There might be papers referenced that will help you better understand and contextualize the paper you’re reading, too.
Review articles provide a summary of multiple research articles to give an overview of the existing research in a particular field, and they can be a great way to get a sense as to what other researchers have been investigating, what questions have not yet been answered, and where the research might be headed.
These reviews are often done systematically, with criteria for inclusion and methodology spelled out, and they too will often be peer-reviewed. Just be aware of the date of the review, since cannabis science is moving at a fast clip. If it’s more than a couple of years old, be sure to seek out what’s happened since.
By no means an exhaustive list, here are a few options for easy search and access to research papers:
- PubMed: A searchable database of over 33 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full text content.
- Sci-Hub: A shadow library that offers access to over 88 million full-text articles, bypassing paywalls in an effort to make scientific knowledge freely accessible to all.
- Google Scholar: A search tool from Google for scholarly literature and academic resources.
Dr. Sulak also offers free monthly webinars through his education platform and online community at Healer.com, where he covers new cannabis research with other doctors, researchers, and experts in the field.
When it comes to cannabis research, Dr. Sulak recommends getting “comfortable with maybe not getting the answer, but getting one step further in your journey towards the answer.”
Carl Sagan once said, “We can judge our progress by the courage of our questions and the depth of our answers, our willingness to embrace what is true rather than what feels good.”
So, be curious. There is so much joy to be had in better understanding the cannabis plant.
