The Design of Experiments: The World of Theory

In the more confined and rigorous world of scientific analysis, one works to develop “theory” rather than “decision-making models.” (You, as graduate students who are interested in applied problem-solving, will have to be at home in both worlds).

There are a number of terms which are used to describe steps in the theory-building process:

Concepts: “The function of scientific concepts is to mark the categories which tell us more about our subject matter than any other categorical sets. Whether a concept is useful depends on the use we want to put it to.” Abraham Kaplan
Axioms: Propositions assumed to be true

Axioms are causal assertions. They are untestable (i.e., they are “asserted”) because it is impossible to control all the relevant variables.
An axiom implies a direct causal link among variables.
Axioms assert a sequence over time by which two things change together (“covariance”)

Laws:

Laws are propositions which have been proven to be true.
A successful inquiry of particulars establishes “fact”; an inquiry of general statements establishes a “law.”

Theories

Theory must interrelate propositions, 2 or more at a time.
Ideally, a theory is a minimal set of propositions (axioms) from which all other propositions may be derived by logic. The reason for this is that the measured covariance between variables that are linked through other variables will be weak, because masked through the intervening variables. So a theory should have as few variables as possible. This is called the “Princple of Parsimony.”
A general theory is a small number of abstract variables, linked to measurable variables.
An auxiliary theory links abstract theory to substantive problems.

Prediction

Short-run prediction does not need theory
If any major parameter varies, theoretical explanation will be needed to make prediction.
Only a small fraction of predictor variables are also explanatory (parsimony, remember).

Complexity: There are a number of ways to make a theory more complex (and less abstract and more similar to the world it is mapping):

Add more variables
More complex forms of relationship (nonlinear, nonadditive joint effects)
Dynamic processes (time paths, feedbacks, cycles)
More realistic assumptions (unexplained variation, measurement error)

There are also a few rules which are commonly used to frame a theory:

· Empirical verification: The theory must correspond with observed reality

· Operational definition: Define the terms used in the theory by the operations involved in manipulating or observing the phenomena to which they refer.

· Controlled observation: Outcomes must be observed under different values of input and when all other variables can be discounted as possible causes of any change in outcome (these are sometimes referred to as “contrafactual”).

· Statistical generalization: Theory is tested on a random sample from the set of conditions to which you wish to generalize.

· Empirical confirmation: Consistency with other verified statements increases the probability of truth.

Finally, there is a logic to linking data to the propositions which they measure.

· The first principle is simplicity—the theory should have a small number of abstract variables linked in explicit ways.

· The second is validity—of which there are several forms.

o Construct validity is the accuracy of the operationalization—do the data in fact capture the phenomena to which they refer?

o Internal validity is the accuracy with which the causal linkages between the propositions is captured in the theory—does it hold together?

o External validity is the accuracy with which the subset of the world on which the theory was tested reflects the entire world to which the theory applies—does the theory capture the whole domain to which it applies?

o Reliability is the accuracy with which the results of the theory can be replicated time after time—how much variability is there in the fit with the whole domain across time?

Next Section

Back to Syllabus

609