What is Evidence? Part 1: Randomised controlled trials
Randomised Controlled Trials (RCT) are one of the most important methods of testing medicine, diets, exercises, and much more. We use it on animals as well as humans to figure out if the thing we are testing works. Basically, most things that apply to clinical settings in regards to health. Their results can be an excellent form of evidence if the study is conducted on a large number of people and over a relatively long period. Most importantly, it’s great to know if you’re debating a claim with someone and they start throwing clinical studies at you. This post will tell you how to investigate what they’re claiming and figure out for yourself if they’re full of crap…or not.
I’ve mentioned RCTs in several of my posts already. I’ve usually given a brief description of what one is, but since I can’t go into detail, I thought I’d go into a bit more detail here. Of course, I’m oversimplifying the process, but it’s to keep everything simple and easy to understand. I’ll link other sources where you can go in depth.
First, a little story
One can take a pretty long ride back through history to find some form of a clinical trial being tried out millennia ago. I think it was even mentioned in the Bible! But let’s not pretend that gives it any more credibility, okay?
The earliest record of a real-life RCT was back in the 1700s. The primary way of getting around the world was by ship. Unfortunately, on long voyages, many of the sailors were getting very unwell and even dying. Their teeth would fall out, old wounds would reopen, and their joints would swell up painfully.
The disease was called scurvy.
James Lind, a young surgeon, looked into the matter. He had a hypothesis that acids could help the condition. He wasn’t the first to think so since it was known that citrus fruits helped the sailors with such symptoms. But it hadn’t become widespread for some reason.
So, he decided to go along on a voyage. After about two months at sea, the sailors started to show symptoms of scurvy. He divided the twelve sailors into six groups of two each. They all were given the same diet, but with one difference. One group was given cider, the second got twenty-five drops of “elixir of vitriol”. Don’t be fooled by that fancy name. It was diluted sulphuric acid. The third for vinegar, the fourth got seawater, the fifth got two oranges and one lemon, and the final group got a spicy paste and a drink made with barley water.
The test ran for six days before the citrus fruits ran out. They were quite expensive at the time. Fortunately, the two sailors, in the group with the oranges and a lemon, had recovered almost to full strength by then. The only other ones that had seen some benefit was the first group with the cider, but only by a bit. No one else got better.
He published his work called The Treatise of the Scurvy, but it initially got utterly ignored. However, over time, as more officials of the navy saw his idea worked, lime juice started becoming more of a staple on the navy ships of the time and scurvy rates dropped among sailors.
Starting an RCT
It all starts with an idea - a hypothesis. This is probably the easiest part and the place where most science begins. You see the world around you, you recognise trends and wonder if there’s a connection. Just like James Lind did. There had been observations and associations had been made that citrus fruits helped the symptoms of scurvy. He wanted to figure it out for sure. Was it just a coincidence, a correlation, or causation? Is it just the lemon, anything sour, or something else?
So, say there is a new medicine. A lab somewhere has had some promising findings in basic research on different chemicals they’re looking at to fight a particular disease. They’ve just put the bacteria causing the illness into a Petri-dish and added different chemicals to see what happens. And voila! It kills the bacteria!
But that doesn't mean all that much in the big scheme of things. Is the chemical they used poisonous to other living things? Or is it well tolerated? Does it have side effects? How will it get to the disease in a living subject? Just a few good questions are asked, and you have the premise for a good experiment.
The first stage is for this medicine to go through animal trials. We do RCTs here too. But let’s make things exciting and presume the rats did well on the meds and it’s time to see if it works in humans also. Note: Just because a trial succeeded in mice or rats, it doesn’t mean they’ll have the same effect on humans.
Randomised
We take a sample of the population now. We select a bunch of people with this disease, the more we can include, the better.
Why? Because everyone is different. If we want to see if this medicine works on most people, we need to take as many people as we can to see the trends more clearly.
Now, these people should ideally be selected across races, ages and genders. Of course, if the researcher wants to focus on a particular subset of the population, they need to mention it clearly in the paper.
Now, this bunch of people get randomly assigned to two or more groups. For the sake of simplicity, let’s stick with two.
Controlled
One group, called the control group, gets a placebo of some sort that looks almost identical to the actual medicine. Think of them as the baseline reading. The real medicine is given to the other group.
Placebos can just be sugar pills, flavoured syrups, fake injections, sham surgeries or any other intervention that only resembles the treatment being tested but isn’t meant to work. The point is to make the placebo identical to the treatment being tested except it excludes the active ingredients.
Depending on the context of the study, the control group could also be given nothing. Another group could also be added in other settings where a new drug is being compared to an already well-established medicine with well-documented results.
Blinded and Double Blinded
Whenever a test is run, it’s essential that it be blinded; otherwise, the biases of the candidates and the scientists could skew the results. Let’s first sort out the terms.
A blinded trial is when the candidates in the experiment don’t know if they’re getting the real medicine or a placebo. But the scientists and analysts do.
In a double-blinded trial, neither the candidates nor those conducting the test know who got the medicine, and who the placebo. But the analysts know.
A triple blinded trial is where no one involved in the experiment knows who gets which medicine.
If you think about it, if the patients have a preconceived notion that a medicine will work, they may feel better for a bit or claim to feel better even if they aren’t to show the drug in a better light.
If the scientists conducting the study know which group gets the medicine and they have a bias towards the medication, they may record more positive results and fewer negative ones.
This may not even be a conscious effort, but it can happen subconsciously. That’s how easy our brains are to fool. And that defeats the purpose of an RCT which is to find the results without any influences from biases. The example I wrote about in my article on homoeopathy on a study by Jacque Benveniste illustrates this quite well.
Analysis
So now that the test has been conducted, the data is analysed to check the results. A trial could be unblinded at this stage or the next one. Unblinding is what it sounds like. Basically, we lift the veil and reveal which patient has been given which treatment.
If the results of improvement are similar to placebo, we say the medicine is no more effective than a placebo. Check out the homeopathy article for that example.
Beyond that, the effect can be measured. If the results are positive with minimal side effects, the chances are good that there will be further trials. And make no mistake, there will be many, many more trials – ranging in demographics, duration and number of patients – that have to be conducted before a drug can hit the pharmacy shelves. Each effect and side effect has to be recorded meticulously and repeatedly so the doctor can compare the benefits to the risks to give their patients the best prescription possible.
Problems with RCTs
Of course, it’s not a perfect system, but it’s quite indispensable as a part of the process in modern medicine.
The problems arise when the blinding isn’t thorough so biases creep in, or if a paper doesn’t get published if the results aren’t in favour of the medicine. Negative results are being suppressed in many cases. There can also be problems in conflicts of interest, which basically means the people funding the study want the scientists to publish only a positive result regardless of the real findings.
Another problem could be the manipulation of data to get a particular p-value - a measure of statistical significance. This term is best explained by Steven Novella, an academic clinical neurologist at Yale University School of Medicine, host of The Skeptic’s Guide to the Universe, and author of a book by the same name:
The primary method for determining significance is the P-value – a measure of the probability that the results obtained would deviate as much as they do or more from a null result if the null hypothesis were true. This is not the same as the probability that the hypothesis is false, but it is often treated that way. Also, studies often assign a cutoff for “significance” (usually a p-value of 0.05) and if the p-value is equal to or less than the cutoff, the results are significant, if not then the study is negative.
He goes on to explain p-hacking:
This is the practice of tweaking the choices a researcher makes in terms of how to gather and analyse data in order to push the results over the magic line of significance. Many researchers admit to behaviour that amounts to p-hacking. Further, when published results are analysed, they tend to suspiciously cluster around the cutoff for statistical significance.
How to make the most of RCTs
RCTs are an indispensable tool to discover the real effects of medicines, diets and lots more. This is a basic form of preliminary evidence. Many of the flaws can even be overcome if other scientists review the work (peer review) and analyse the results to check for mistakes or shoddy calculations. Other researchers could try replicating the experiment to see if it works or not.
But what does this mean for you?
Finding evidence for claims through RCTs can be a massive help in finding the facts. But it’s not easy for most laypeople to understand RCTs and what they imply. There’s a lot of jargon thrown in usually, and the statistical references can fly over your head if you’re not well versed with the subject. Worse yet, many clinical studies are stuck behind a paywall and meant for academic purposes only. Then you’ll have to rely on other sources, which I’ll mention a bit later.
An excellent site for finding whole RCTs for free is Pubmed. I’m sure you’ve heard of it. Pubmed has become an excellent repository for a lot of studies across a wide range of fields - over 29 million actually. But just because it’s on Pubmed doesn’t mean its a well-designed study.
That’s why you need to be able to evaluate them for yourself to figure out if a study is legit or not. Here are a few things you can do to start off with. Consider this a noob’s guide to figuring out RCTs. Ask the following questions of the RCT and figure out the answers:
How large was the study? The higher the number of participants, the more reliable and universally applicable a study is.
How long was the study conducted? Similar to the earlier point, the longer a study is done, the more information we have of long term effects of the experiment. For example, if it’s a diet, we can measure metabolic impacts over the long term and how sustainable the diet is.
How many people left? Seeing if any of the participants left the experiment can also have hidden clues. It’s not unusual for people to drop out from an experiment, but do the researchers report why they left? If they do, it could say a lot about unforeseen problems or side-effects of the thing being tested. If not, something fishy could be going on.
Who participated? The population tested could also have ramifications on how relevant the study is to you. Was it conducted only on seasoned athletes or diabetics or males aged over 65 with a history of prostate cancer? Are any of these demographics similar to yours? If not, it’s possible the study isn’t relevant for other groups. There is a chance there is a broader relevance, but that has to be clearly stated in the report.
How were the groups split up? Understanding the methodology of the study can also prove vital. This will take a bit of training though, but it’s worth delving into. I’ll mention books you can read to help you here in the Further Reading section.
What were the results and how were they interpreted? This is where the evidence becomes more explicit. Most of this is usually written in much clearer language and is relatively more easy to understand. But look out for terms like, “further research is required”. This is the researchers clearly stating their study is still preliminary and more needs to be done to really understand how everything is working.
Pro tip #1
If you can only find studies trying to evaluate the safety of a particular chemical, there is a good chance that the product is alternative medicine. This is because many alternative and complementary meds (CAMs) try to go through the US Food & Drug Administration’s guideline loophole that implies you can sell unproven treatments in the form of supplements provided you prove they are safe for human consumption. Clever, right?
Pro tip #2
Many people will possibly quote studies when you’re in a discussion regarding a specific claim. Use what you’ve learned from this article to investigate further. If you find the research was done on mice or earthworms or something, you can safely respond with a “this is interesting, but since it’s not on humans, nobody can use this as proof that your claim is valid.''
In other words, “you’re full of shit”.
Noob tip
If you can’t figure out what’s going on, or if you’re just starting out and need guidance, try and look for the references to a study or a claim through reliable sources like science magazines, WHO, US and UK government websites, NASA, or even Wikipedia. I’m going to be writing in more detail about Wikipedia in the next article in this series, but until then, just know that it’s a significant first step to figuring out any topic. Plus, all the citations are listed so you can go deeper into a subject by checking those out.
Some great websites and blogs you should also search on are Science Based Medicine, Quackwatch, Skeptic, Neurologica and Snopes.
Conclusion
If you are trying to figure out whether a claim is valid or not, RCTs are definitely useful. But it’s just one tool in a toolbox of instruments that can help you dig deeper into a topic to understand the facts, which we will continue to discuss here on Rationable.
Finding the facts is not an easy task. I have been teaching myself these tools for over a decade now. I’ve picked them up from scientists and other science communicators and sceptics who use them regularly. I started at a point where only the abstract of a study was all I could understand. Now, I’m no scientist, but a lot of other pieces have fallen into place.
This is one of the main reasons Rationable exists. I wanted to show you that scientific thinking and evidence is something laypersons can learn and use too. You don’t have to be a scientist to test people’s claims and understand science. I’ve taught myself this process and it has been an incredibly empowering experience, not to mention awe-inspiring.
As I said, finding the truth is hard. And that’s why so many people don’t do it. So many of us believe a Whatsapp forward from a close relative or a friend or sibling we respect. They become our trusted sources because we trust them. But are they objectively reliable sources of information? No! For that matter, neither am I. I am just as biased as anyone else. I get my news from secondary and tertiary sources. Each and every one of us is biased, and we naturally want to agree with the information that fits those biases and our belief systems. We need to understand this, accept it and actively work against those instincts to find evidence that could contradict them. At the end of the day, we need to follow the evidence, not our biases.
That’s why I want you to fact-check everything I say. The claims I make are not made from my expertise but rather from all the sources I get them from. And those too are secondary sources most of the time. But I link you back to them in the article and in recommended reading at the end of each post so that you can go back and check them out and check where they got their facts from.
Randomised Controlled Trials, though are an effort to minimise our biases. Making them blinded and subjecting them to peer review makes them even more reliable. James Lind fashioned his experiment based purely on common sense, and it worked. Now, that has become a relatively crude experiment as we have continued to refine the process to make the results progressively more objective, free from bias and ethical. I’m quite sure the process will continue to be refined to minimise the problems it faces now.
Have I got something wrong? Did I miss a detail? Let me know in the comments.
If you enjoy this content, please bookmark it, subscribe to the RSS feed and share it with your friends. Your support keeps Rationable going.
References & Further Reading
If you want to buy the books, I’d appreciate it if you could use the links I’ve added below. I’ll get a small commission which supports Rationable and everything that it does.