[This article originally appeared in the
June 2001
issue of Northwest Runner
magazine.]
Last month, we took a look at some critical questions one should ask when trying to make sense of training methods based on scientific research. As it turns out, though, there are so many important questions that I wasn't able to cram them all into one column. Here, then, is part two: further issues to be considered when you encounter someone who says that a particular approach to training or nutrition is "supported by research." As before, my overall point is that you shouldn't believe everything you read or hear -- even when the writer/speaker claims to have science on his (or her) side.
1. Were lots of things changing simultaneously during the study? When multiple experimental variables are changing simultaneously, sorting out causes and effects can be difficult if not impossible. This conundrum is exemplified by the paper of Mizuno et al. (Journal of Applied Physiology 68: 496-502, 1990), who attempted to determine the impact of altitude training on elite cross-country skiers. The skiers were tested before and after a two-week altitude training camp -- which is all fine and good, except that they did tons of cross-training while at sea level and mostly skied while at altitude. So, were the pre- to post-camp changes in the skiers' muscles due to the change in elevation, or to the change in training? We'll never know for sure.
2. Have the data been reported accurately? We can usually count on researchers to get their numbers right, but errors do occur. The central finding of a 1999 study by Rodriguez et al. (Medicine and Science in Sports and Exercise 31: 264-8, 1999) was that hanging out in a hypobaric chamber could increase endurance time in a treadmill test (administered outside the chamber) by 3.9%, yet the raw data reported in the paper show that the average endurance time increased from 15.6 minutes to 16.5 minutes, an increase of 5.8%. A minor point, really. But then Dr. Phil Maffetone, writing about this study in the Road Runners Club of America newsletter FootNotes, equated this 4-6% gain in treadmill endurance time to a 4-6% drop in one's race times, which is just plain silly. If you're a 40-minute 10K runner, for example, a 5% improvement in endurance time would mean that you can now hold your 10K pace for 2 minutes longer than you could previously. This is noteworthy progress, but it is obviously not equivalent to cutting 2 minutes off your best 10K time.
3. Are the differences between groups statistically significant? In any study involving two or more groups, the groups are unlikely to turn out exactly the same. But are the apparent differences between groups just due to random chance, or are they real? This question is best addressed using statistical tests; differences that are "statistically significant" probably aren't the result of chance alone, whereas non-significant differences may be entirely due to random variation. Unfortunately, scientists and coaches sometimes overemphasize the importance of minor, non-significant differences. For instance, a recent study by van Loon et al. (American Journal of Clinical Nutrition 72: 106-11, 2000) showed that a high-carbohydrate drink was approximately as effective as a carbohydrate-protein mixture in replenishing muscle glycogen stores after a hard workout. However, the slight, non-significant advantage of the high-carbohydrate fuel over the carb-protein mix led Running Research News to announce, "High-carb eating wins TKO over carb-protein combo in recent glycogen-boosting tests" (Running Research News 16(8): 7-9, 2000). To make this claim based upon a result that may be due to chance is inappropriate and misleading.
4. Has a correlation been confused with causality? Consider the following statement: "Research by renowned physiologist and coach Jack Daniels (Medicine and Science in Sports 10: 200-3, 1978) has shown that people with large feet are better runners than people with small feet. This may be due to the fact that large, powerful feet allow for a strong 'push-off' with each stride." It's true that, in the study cited, those with who ran faster tended to have bigger feet, but that's because the subjects of the investigation were boys aged 10 to 18! Not surprisingly, older boys have larger feet and run faster than young boys. In this example, as in the previous one, the original research article is fine, but an outside observer has misinterpreted the data. The message here is that we shouldn't assume trait X causes outcome Y simply because X and Y tend to change in parallel.
5. Are the conclusions based on questionable extrapolations from the data? In the April 2001 edition of Northwest Runner, Gatorade ran an ad which states, "Research shows your body needs at least 40 oz. of fluid every hour or your performance could suffer." In support of this assertion, it cites the American College of Sports Medicine's guidelines on fluid replacement (Medicine and Science in Sports and Exercise 28(1): i-vii, 1996). So what do those guidelines really say? Well, the data indicate that, in some situations, water loss due to perspiration can exceed 40 ounces (1.2 liters) per hour. However, of the 92 references cited in the ACSM's position stand, not one demonstrates empirically that gulping 40 or more ounces per hour aids performance more than a more modest rehydration regimen. Now for the case of a large person running an ultramarathon in the heat, Gatorade may be right: this person may need to imbibe 40 ounces an hour to maintain his/her body weight and competitive edge. For someone running a half-marathon in cool weather, on the other hand, hydration per se seems unlikely to improve performance. Furthermore, there is such a thing as drinking too much. Consuming more fluid than is lost in sweat can sometimes lead to hyponatremia, a condition in which sodium levels in the blood become dangerously low (Speedy et al., Emergency Medicine 13: 17-27, 2001).
6. Do the results of this study agree with those of other studies? There's always a danger in getting one's information from a single study, even if that study seems well-designed and well-executed. The above-mentioned Dr. Maffetone -- coach of such triathletes as Mark Allen and Mike Pigg -- would do well to keep this point in mind. Maffetone's website used to include a "Science and Medicine Review" which cited the work of Kinscherf et al. (Journal of Molecular Medicine 74: 393-400, 1996). According to Maffetone, Kinscherf et al. demonstrated that "healthy people who performed anaerobic exercise three times a week for one hour over a four- to eight-week period . . . showed unhealthy states, including low amino acids (glutamine and cysteine)." What Maffetone fails to mention is that the Kinscherf et al. article actually includes data from two separate studies. In one study, the amount of glutamine and cystine (not cysteine) in the blood did indeed go down following "anaerobic" training (sprinting and weight-lifting). However, in the other study, the levels of these amino acids went up after anaerobic training. Obviously, the consideration of multiple studies can yield a very different perspective than one might get from looking at only one. And since a single research report may not represent the consensus of the research community, those who base their opinions on one or two studies do so at their own peril.
7. Are the relevant sources identified? People who are serious about applying scientific research to athletic training should be able to tell you exactly where they're getting their information. If they can't, it probably means that they haven't taken a critical firsthand look at the data in question, in which case their understanding of the data may be limited. But this isn't just an issue of scientific knowledge; it's also one of coaching philosophy. I believe that athletes should have access to the scientific evidence upon which their coaches base their recommendations. After all, if a coach has considered the evidence and has used it to design a sensible training program, what has he/she got to hide? The alternative, of course, is for coaches to adopt a "just trust me" attitude. In my opinion, however, it is the athletes who must be trusted -- trusted with the freedom to think about and discuss what the scientific literature really means.