Notes on Experimental Design
Components of Design
- Questions
- Objectives
- Consequences (intended & unintended)
- Propositions (Theory)
- Prove the obvious
- "Variables"
- independent
- intervening
- dependent
- "Explained" & "unexplained" variation
- Units of Analysis (Data)
- Operational definition
- behavioral performance
- measurement (nominal, ordinal, interval, ratio)
- Segmented population
- Data collection issues
- accuracy (avoid shifts over time/across uses)
- completeness (all relevant population segments included)
- comparability (same definitions applied in similar settings)
- problem of "the volunteer"
- Logic Linking Data to Propositions
- Simplicity: small number of abstract variables linking in explicit way
- Validity
- construct (operationalization)
- internal (causal linkages)
- external (domain)
- reliability (replicability)
- Criteria for Interpretting Findings
- Control ("contrafactual")
- Exploratory data analysis (smooth [model fit] + rough [residuals])
- resistant measures (median)
- residuals (stem & leaf display)
- re-expression (transformation)
- revelation (graphic display)
- Effectiveness vs. efficiency (cost/unit)
Types of Experimental Design
- Ideal: Experimental Model
- Treatment & control groups
- Pre-test & post-test
- Random assignment
- Asks "what" & "how"--no values, no "why"
- Quasi-Experimental Designs
- Threats to validity
- internal
- history (specific events occuring between first & second look)
- maturation (process within respondent because of time passing)
- testing (prior experience affects subsequent behavior)
- instrumentation (change in calibration or observers)
- regression (groups selected for extreme scores shift to middle)
- selection (bias in original assignment)
- mortality (differential loss of respondents)
- interaction
- External
- testing interaction (pretest sensitizes respondent to IV)
- selection interaction ("Hawthorne Effect")
- reactive (generalization from controlled to uncontrolled setting)
- multiple-effect interference (causal factors interfere with each other)
Solutions to Validity Threats
- Experimental
- Pre-test/Post-test Control
- Solomon 4-group Design
- Post-test Only Control
- Quasi-Experimental
- Time Series
- Equivalent time samples
- Equivalent materials
- Nonequivalent control groups
- Counterbalanced Designs
- Separate sample pretest/posttest
- Separate sample pretest/posttest control
- Multiple time series
- Institutional cycle design (almost grounded theory)
Observational Design
- Issues
- Uncontrolled
- Causal Ordering (A then B, therefore A caused B)
- Descriptive--"what" & "who"
- Precursor for controlled design (grounded theory)
- Case Study
- The Method
- use multiple sources of evidence (Campbell's "triangulation")
- determine data base
- chain of evidence (necessary & sufficient)
- Analysis of case data
- pattern-matching
- explanation-building
- time-series analysis
© 1996 A.J.Filipovitch
Revised 19 September 96