Self-experimentation seems to be better for generating ideas than more conventional ways of collecting data, just as exploratory data analysis seems to be better for generating ideas than more conventional ways of analyzing data. -- Seth Roberts (short paper)
Seth Roberts, author of the Shangri-La Diet has a four part series on self-experimentation. (See part 1, part 2, part 3, and part 4.) In these posts, he seperates his brilliant advice into 7 key concepts. Here's a summary of what he wrote:
1. Do something. I started the long-term self-experimentation that led to my paper because I didn’t want to wake up too early for the rest of my life. I expected my little self-experiments to fail, and they did fail, but I didn’t realize that I would slowly learn from failure. I learned how to record my data, for instance, and how to analyze it. The effect of that learning was that my self-experimentation got better and better and after many years of failure I got somewhere. I think American culture teaches that success is good and failure bad, but the truth for scientists is that failure is good in the sense that you learn from your mistakes.
2. Keep doing something. I learned the value of drudgery. The research took many years. After my initial failures I continued not because I could see I was learning stuff — the learning was too slow to be perceptible — but for the same reason I started: I didn’t want to wake up early for the rest of my life.
There were many possibilities and no alternative to simply testing them one by one. (More complex experimental designs, such as factorial designs, were impractical.) There was nothing intellectually wonderful about it.
3. Be minimal. In other words, do the easiest, simplest thing that will that will tell you something important and new, that will provide significant progress.
Almost all proposed research I hear about breaks this rule, which is fascinating in a train-wreck kind of way. I have never seen a book about research design that makes this point. As a result, I suspect that books about research design are often counter-productive: The student would have been better off if he or she hadn’t read them. The textbook teaches this or that complication to people who can barely do basic stuff. The poor student wastes time using complex designs that fail in cases where a simpler design would have succeeded.
4. There are serious defects in the way science is usually done. I found a new and powerful way of losing weight — yet I’m an outsider to that area. Although obesity is a huge problem, and hundreds of millions of dollars go into obesity research every year, I was completely outside that group of people and resources. If science is being done properly, there should be a relation between input and output — the more input, the more output. That failed here.
Professional obesity researchers, given vast input, failed to discover this; whereas I, given zero input, managed to do so. You might say this was a weird fluke except the same thing happened again with mood: I discovered a powerful way of changing mood, even though I was an outsider to the study of mood. Depression is a huge problem, vast resources go into trying to do something about it.
What the serious defects are has no simple answer. After the next lesson learned I’ll try to explain what I think is wrong.
5. There are serious strengths in the way science is usually done. I relied heavily on conventional science and could never have gotten where I did without it. I didn’t just use conventional scientific tools; I also used self-experimentation, which is unconventional. But self-experimentation alone wouldn’t have gotten very far, I’m sure.
To say that science is glorified common sense has a lot of truth to it. To say that science is a collection of methods to help us understand and control the world also has a lot of truth to it. But science is far more than a collection of tools; it is a whole community and culture, with beliefs as well as tools. Like any culture, many of its beliefs are based on faith.
Conventional scientific methods could go part of the way toward finding the Shangri-La Diet; but they could not go all the way. Other techniques were needed — very simple ones. So conventional science never found it.
6. Curiosity helps — because it provides a wide range of knowledge. Pasteur made a similar point when he said luck favors “the prepared mind” by which he meant the well-stocked mind. To come up with my theory of weight control you needed to know both obesity research and animal learning because the theory is based on basic facts about weight control and basic facts about Pavlovian conditioning. I knew the weight control facts because I had taught introductory psychology and lectured on weight control. I knew the basic facts about Pavlovian conditioning because my graduate training was in animal learning. It was unusual to know both sets of facts.
7. Publish in open-access journals. Because my long self-experimentation paper was published in an open-access journal, anybody could read it within minutes. My friend Andrew Gelman blogged about it, which caused Alex Tabarrok at Marginal Revolution to mention it. This brought it to the attention of Stephen Dubner, who with Steven Levitt wrote about it in their Freakonomics column in the New York Times. That led to a contract to write two books — one about weight loss, the other about self-experimentation in general. That anyone could download my paper made it spread much faster. In the old days, with photocopies and libraries and mailed reprints . . . no talk tonight.
A summing-up, if you want to figure something out via data collection:
- Do something. Don’t give up before starting.
- Keep doing something. Science is more drudgery than scientists usually say.
- Be minimal.
- Use scientific tools (e.g., graphs), but don’t listen to scientists who say don’t do X or Y.
- Post your results.
Seth Roberts truly is an original thinker. Unfortunately, I missed him when he spoke at GMU last semester. I'm intrigued by his ideas about using self-experimentation as a way of discovering new knowledge and want to learn more about the methodology behind his concepts.
You can visit Seth's blog and homepage to learn more about him and his unique ideas.
Read more about Seth here, here, here, and here.
See my previous post on Seth Roberts and self-experimentation here.
No comments:
Post a Comment