Guide to Criticizing Research:

Theory

Do the implications for this theory matter?

How is this theory related to others?

Based on what criteria can it be differentiated from others?

Does this differentiation matter?

What are the assumptions of the theory? Are they explicit?

Does the theory make sense? In other words, based on what prior evidence or set of logic is the theory based? Is either the logic or the prior evidence faulty?

Can the null hypothesis be accepted? Based on which principles can the null hypothesis be accepted? Are those principles explicit?

What is the contribution of the theory as connected with other theories?

What theories can be inferred if this theory is correct?

Are there any other theories that must be modified if this theory is correct?

Units of analysis

Considering the unit of analysis: what are other possible units – go up on level in aggregation and down on level of aggregation. Remember that going down one level in aggregation on the individual level could be akin to an interaction in terms of studying units of individual-situation. You could also organize individuals in many different kinds of groups: race, SES, education, friends, some form of larger social network, geographically. This research might be conducted in terms of aggregate groups, or modeled as an interaction between groups and individuals.

If the units of the analysis changed, how would this affect the findings?

Consider the variation across units. Does the research include all empirical variation? What about hypothetical variation? How is this inclusion or exclusion likely to affect the results?

Based on which criteria were the units selected? Remember that maximizing variance on independent variables is a good technique and selecting on the dependent variable is most often a bad technique.

Are the units homogeneous? This means that for a one unit change on the independent variable of one unit, there will be equal expected change on the dependent variables across all units. Examples where this might not be the case is when the units are nebulous and can be counted at varying degrees of specificity.

Measurement

For a concept to be a concept, it must be unidimensional, with well-defined extremes of a continuum. What are the hypothetical as well as empirically defined extremes of the continua of each of the variables?

Is the measurement of the variables valid on its face?

Thinking about re-measuring the variables, would the results be the same under different conditions?

If the concept is measured on a scale, are the items internally consistent?

Concepts must be fully specified, mutually exclusive and exhaustive.

Methodology

Is the methodology undertaken the most conservative test of rejecting the null hypothesis?

Does the methodology allow for interpretable results?

Inference

Does the inference match the test?

Is there anything about the design that is weak in differentiating inferring results to be compatible with one theory, as opposed to another?

What are the implications of the research? Do these implications match the stated purpose of the research?

How do the findings fit into other literature?