Title
21st Century Research Protocols
Team Members
Slime 1 and Slime 2 of SLIME MOLD TIME MOLD
Short summary of your improvement idea:
Develop new forms of scientific studies that will do for the 21st century what the randomized controlled trial did for the 20th.
What is the existing target protocol you are hoping to improve or enhance? Eg: hand-washing, traffic system, connector standards, carbon trading.
Scientific research today relies on a single main protocol, experiments with control groups and random assignment, which in medical contexts is usually called a randomized controlled trial (RCT). This is a powerful invention for detecting population-level differences across treatments or conditions.
However, the RCT is only one tool, and like all tools, it has specific limitations. When applied to situations where population-level differences aren’t the question of interest, or where there is substantial heterogeneity of treatment, especially if the heterogeneity is the main topic of interest, the RCT is a poor tool and often gives incoherent answers.
Put simply, if people respond to a treatment in very different ways, an RCT will give results that are confusing instead of clarifying. If some people have a strong positive response to treatment and some people have no response at all, the RCT will distill this into the conclusion that there is a mild positive response to treatment, even if no individual participant has a mild positive response!
In addition, RCTs are inefficient. They can test only one hypothesis at a time per several dozen or several hundred participants. They don’t take advantage of modern cheap computation and search algorithms. For example, in the 1980s there was some interest in N=1 experiments for patients with rare cancers. This was difficult in the 1980s because of the limited access to computers, even at research universities. But today you could run the same program on your cell phone a hundred times over. We want to make use of these new insights and capabilities.
Similar ideas apply outside of study design. Until recently it was hard to get expert review of your work, and often you wouldn’t be able to find the experts you need, or even know what experts were relevant to ask. But as Linux demonstrated in the world of software, with the internet you can get massive parallel review of your work.
What is the core idea or insight about potential improvement you want to pursue?
A new century requires new scientific protocols. The 21st century is an era where communication is prolific and computation is cheap, and we will harness this power to advance the creation of knowledge.
Since its early days, science has been based on doing experiments and sharing results. Researchers collect data, develop theories, and discuss them with other interested parties. New technology has made it easier to do experiments and share results. But like the printing press, which was invented in 1440 but didn’t lead to the Protestant Reformation until 1517, the internet hasn’t yet been fully leveraged.
Here are four protocols we’re developing, that show exceptional promise:
-
The idea of N = 1 experiments / self-experiments has been around for a while, and there are some famous case studies like Nobel Laureate Barry Marshall’s self-administration of H. Pylori to demonstrate its role in stomach ulcers and stomach cancer, but this protocol has yet to reach its full potential. We hope to improve this method, especially for individuals with chronic illness/conditions that bamboozle the doctors. We also want to address common issues we see with study design to help individuals make their self-experiments as informative as possible. We have already begun a blog series about N = 1 / N of Small studies, and will continue writing about this topic and advising other researchers.
-
The Community Trial is a new protocol that blurs the line between participant and researcher. In these trials, an organizer makes a post providing guidelines and a template for people to share their data. Participants then collect their own data and send it with the organizer, who compiles and analyzes the results, sharing the anonymized data in a public repository. Because data collection is self-driven, participants can choose to measure additional variables, participate in the study for longer than requested, and generally take an active role in the study design. Unlike most RCTs, community trials allow for rolling signups, and could be developed into a new class of studies that run continuously, with permanently open signups and an ever-growing database of results with a public dashboard for analysis. We first tested this with the Potato Diet Community Trial (announcement, results), where 209 people enrolled in a study of an all-potato diet and the 64 people who completed 4 weeks lost an average of 10.6 lbs.
-
We recently invented a protocol called the Riff Trial. This protocol takes a treatment or intervention which is already somewhat successful and recruits participants to self-assign to a close variation on the treatment. Each variation is then tested, and the results reported back to the organizers. This uses the power of parallel search to quickly test possible boundary conditions, and possible variations that might improve upon the original. Since each variation is different, and future signups can make use of successful results, this can also generate improvements based on the power of evolution. We are currently testing this protocol for the first time in the SMTM Potato Diet Riff Trial, with two rounds of results already reported (Round 1, Round 2). The main discovery so far is that consuming dairy does not seem to be a boundary of the potato diet, as was originally suspected.
-
This summer, we will pilot an untitled new kind of trial focused on broad hypothesis testing of boundary conditions. In our previous studies, participants are hesitant to “bite the bullet” and try variations that stop the effect, which could test the parts of the intervention that are both necessary and sufficient. So we are designing a new protocol that makes testing these boundaries the centerpiece. In this novel approach, participants test an intervention over baseline to confirm that the standard intervention works for them, then are randomized into conditions with variations that are proposed or suspected boundary conditions for the effect (e.g. “The intervention works, but it wouldn’t work if we did X/didn’t do Y.”). By randomly introducing potential blockers, we will be able to learn more about how robust an intervention truly is, and test theories of why it works, since theories will usually make strong predictions about conditions under which an intervention will stop working. This protocol will also help us better understand differences between individuals, and may reveal that certain variations are a boundary condition for some people and not others.
What is your discovery methodology for investigating the current state of the target protocol? Eg: field observation, expert interviews, historical data analysis, failure event analysis
Our main discovery methods are historical, analogical, and tinkering.
We do close reads and analysis of the successful development of past protocols (for example, the scientific innovation around the cure for scurvy).
We develop new scientific protocols by analogy to successful protocols in other areas. For example, self-experiments are somewhat like debugging (programmers in the audience will be familiar with suspicion towards “well, it worked on MY setup” stories), and the riff trial was developed in analogy to evolution.
Finally, we deploy simple versions of these protocols as quickly as possible so that we can tinker with them and benefit from the imagination of nature. This is also somewhat by analogy to hacker development methods and startup concepts like the minimum viable product. We have already developed three original protocols—community trials, riff trials, and our untitled new trial—which we try out as soon as they are ready so people can tinker with them.
In what form will you prototype your improvement idea? Eg: Code, reference design implementation, draft proposal shared with experts for feedback, A/B test of ideas with a test audience, prototype hardware, etc.
Write descriptions of protocols with consideration of their strengths and weaknesses, when to deploy or not deploy, and guidance on how to use them. We will conduct community trials and advise others. All of our work will be published for free online.
How will you field-test your improvement idea? Eg: run a restricted pilot at an event, simulation, workshop, etc.
Run a test of the new kind of trial over the summer, trying to get at least 2,000 participants (preferably more). Write short guides to protocols we’ve already developed and publish them online to make it easier for others to use and develop these study designs further. As time and energy allows, we will also run N=1 studies and Riff Trials, or work with people to run N=1 studies and Riff Trials of their own.
Who will be able to judge the quality of your output? Ideally name a few suitable judges.
Emmett Shear
Alexandra Elbakyan
Balaji Srinivasan
Scott Alexander (Astral Codex Ten)
Gwern
Future Generations 
How will you publish and evangelize your improvement idea? Eg: Submit proposal to a standards body, publish open-source code, produce and release a software development kit etc.
We will run our own studies and create open-source reference materials for 21st century researchers. We will release all new study protocols as development kits with guides, templates and code for data collection, analysis, and publication. We will offer advising and publish philosophical transactions with interested parties
What is the success vision for your idea?
Over the next five years, internet scientists extend our work into a large collection of diverse research protocols far beyond the RCT. They run a huge variety of studies, make some major discoveries, and cure a few diseases. The discoveries are compiled into open-source textbooks, and the insights gained are made into libraries and kits to make the next five years of discoveries even easier.
Five years from now, in the summer of 2029, we pass the milestone of 10 million people participating in internet science studies based on these new designs. Many of these individuals have found a treatment for their chronic condition that allows them to lead happier, more productive lives.
Driven by prolific communication and cheap computation, this is the dawn of a new golden age of science.