• replication = bedrock of science - can same result be observed by different researchers?

• summarize/synthesize what is known about some research question

• degree of relationships/ differences, estimates of population parameters, variability across studies

• plan next steps in research

• narrative literature review

• reviewer gives verbal, somewhat subjective summary and appraisal of studies

• vote counting review

• counting up how many studies showed statistically significant results

• if all/nearly show significance, then conclude some relationship/difference

• if some significant and some not,  then conclude results are mixed or conflicting

• if nearly all nonsignificant, then conclude no relationship/difference

• estrogen replacement therapy and dementia (Yaffe et al.)

• estrogen prescribed for alleviating symptoms of menopause and preventing certain associated conditions (e.g., osteoporosis)

• other health effects? controversy

• odds ratios between estrogen use and dementia from 10 studies

• odds of dementia for estrogen users relative to nonusers

   Study                      OR (95% CI)

A (case-control) n = 120         2.4 (0.7 - 7.8)

B (case-control) n = 213         1.7 (0.4 - 5.9)

C (case-control) n = 340         0.7 (0.4 - 1.6)

D (case-control) n = 260         1.2 (0.5 - 2.6)

E (case-control) n = 227         1.1 (0.6 - 1.8)

F (case-control) n = 165         0.3 (0.1 - 0.8)

G (case-control) n = 680         0.7 (0.4 - 1.2)

H (case-control) n = 304         0.6 (0.3 - 1.2)

I (prospective) n = 1,124         0.5 (0.3 - 0.9)

J (prospective) n = 472           0.5 (0.2 - 1.0)
 

• meta-analysis

• goal: summarize results across studies quantitatively

• focus on magnitude of difference/relationship/association ("effect size")

• statistical significance irrelevant

• procedures

• select studies for inclusion - whether published, substantively meaningful distinctions, quality

• estrogen & dementia - MEDLINE search, bibliographies of articles, experts

• compare or combine results from different studies

• decide whether studies similar enough to combine

• meta-analysis not pooling all of the data

• rather, analysis of separate studies' results

• e.g., what is the average OR, correlation, etc. in these studies?

• e.g., which type of studies have a higher or lower OR, correlation, mean value, etc.?

weighting by sample size common

• estrogen and dementia -

• weighted mean OR for all 10 studies = 0.71

• for 8 case-control retrospective studies = 0.79

• for 2 prospective studies = 0.48

• meta-analysis shows markedly different conclusion than vote- counting approach

• studies show somewhat decreased risk for dementia among estrogen users

• beneficial effect of estrogen not clear because no good randomized experiments, confounding may exist

• other points

 
• interpretation of meta-analytic summaries similar to interpretation of original studies - research design

• summary of observational studies - cannot make causal conclusions

• retrospective vs. prospective

• factors related to opiate addicts' continued opiate use during/after treatment (Brewer et al.)

• how strongly is patients' cocaine use associated with relapse to opiates?

• retrospective - weighted mean r = .28 (4 studies)

• prospective - weighted mean r = -.03 (3 studies)

• summary of experimental studies - can make causal inferences

• depression causes continued use?

• three experiments - opiate addicts randomly assigned to psychotherapy or no psychotherapy

all showed that psychotherapy reduced patients' depression

• all experiments showed psychotherapy increased opiate use

• thus, depression not causally linked to continued opiate use

• meta-analytic summary consistent: depression very weakly correlated with continued opiate use (weighted mean r = .10)

• some problems common to all research - incorrect interpretations, inattention to detail, relying on significance testing

• including biased or flawed original studies - can use screening criteria to exclude or weight/stratify results by study quality

• file drawer problem - not including unpublished studies