| QUESTIONS TO GUIDE YOUR
EVALUATION OF RESEARCH ARTICLES IN LIBRARY AND INFORMATION SCIENCE
Evaluation framework liberally adapted by Marc
Lampson for educational purposes from Jeffrey Katzer, Kenneth H. Cook, &
Wayne W. Crouch, Evaluating Information: A Guide for Users of Social
Science Research, (McGraw Hill, 1998).
A copy of the 1991 edition has been placed
on 24-hour reserve at Odegaard Library.
QUESTIONS USUALLY ANSWERED IN THE BEGINNING
OF THE ARTICLE –
FRAMEWORK QUESTIONS (everything prior to the methods explanation):
Problem (Research Question) Statement
Is the problem (or the research question or questions) stated so
that it can be solved generally, and particularly by the methods
proposed? Does the way the problems or questions are posed restrict the
study or eliminate other explanations or research alternatives?
Definitions
Are definitions sufficiently specific or are they vague or arbitrary
or circular? Are the definitions useful to your understanding of the
article and in the context of the research tradition of LIS?
Literature Review
Is what was included in the literature review relevant, recent,
balanced, sufficiently descriptive? Does the literature review indicate
the author(s) thoroughly considered the implications of what might be
found prior to gathering the data? Is there any reason to suspect that
the literature chosen for review was chosen because it either confirmed
the study’s findings or provided a “straw man” for the study to defeat?
QUESTIONS USUALLY ANSWERED IN THE MIDDLE OF
THE ARTICLE –
METHODS QUESTIONS: The detailed description of how the data were
obtained. This includes techniques for data collection, objects or
subjects studied, design (fixed, flexible, quantitative, qualitative,
experimental, quasi-experimental, case study, etc.), sampling
procedures, construction and use of research instruments.
Observation
Was anything left out? Was there control for possible distortions of
what was observed? When and where were the observations made?
Measurement
Are operational definitions adequate? (Operational definitions are
the methods of defining variables or constructs based on how they are
measured; variables or constructs are the concepts or ideas being
measured in the study.) In other words, does the study measure what it
says it measures? Does the instrument measure as precisely as the
researcher claims? Is there a measure of reliability provided? What is
really being measured – what the author claims or something else?
Control
Were potential rival explanations eliminated in the research design?
Was more than one group studied for comparisons?
Generality
Are there any arguments made for generality and how strong are they?
Did anything happen during the study that made the people studied less
representative?
QUESTIONS USUALLY ANSWERED TOWARD THE END OF
THE ARTICLE -
RESULTS/CONCLUSIONS QUESTIONS. The results/conclusions are what the
researcher discovered and how those discoveries are interpreted. These
discoveries and interpretations include summaries and analyses of the
data, findings from the data, etc.
Fair Summaries
How were the summaries arrived at, computed, chosen (what
alternatives were there), and what was left out?
Interpretations
Are there missing assumptions or indications that counterexamples or
results were excluded? Do the interpretations make sense given the
literature cited and are the interpretations acceptable to you, do they
make sense? Did the study answer the questions originally asked?
Relationships
Was the analysis based on a single variable, did it consider other
possible variables, and are there yet other variables that you can think
of that were not considered? Was a causal relationship stated or
implied, and if so, was it justified?
Explanations
Can you think of other explanations for what is reported in the
study other than the explanation given in the study? Are all causes, or
parts of the cause, considered and reported?
Statistical Inferences
Were proper statistical procedures used and used correctly? Was
chance eliminated as an explanation? Were the findings statistically
significant? Are confidence levels, if used, acceptable? Are the
findings meaningful? Get some expert statistical advice if necessary.
|
