Panacea or Poison: Can Propensity Score Modeling (PSM) Methods Replicate the Results from Randomized Control Trials (RCTs)?

Problem

The problem the study aimed to address: 

To evaluate whether Propensity Score Modeling (PSM) methods can reliably replicate the results of Randomized Controlled Trials (RCTs) in the context of criminal justice research.

General impact on the system and/or public: 

RCTs are often viewed as the "gold standard" for establishing causal inferences but are resource-intensive and impractical in many real-world scenarios. PSM has gained popularity as an alternative, but concerns about its reliability have implications for policy-making and academic rigor.

Research Questions:

1. Can PSM methods accurately replicate the results of RCTs?
2. Which PSM techniques perform best in replicating RCT results?
3. What are the limitations of PSM compared to RCTs in criminal justice research?
    

Method and Analysis

Program Evaluated/Gaps Addressed: 

The study addresses the gap in testing PSM reliability and validity within criminal justice datasets, where the applicability of PSM is still debated.
 

Data and Sample Size: 

• Data from 10 RCT studies, including one quasi-experimental study.
• Average sample size per study: 573 participants.
• Each study required a minimum of 130 participants in treatment and control groups.
   

Analysis Used:

1. Artificial selection bias was introduced into RCT datasets.
2. Seven PSM techniques (1-1 matching, 1-many matching, inverse probability of treatment weighting, stratified weighting, and optimal pairs matching) were applied to remove bias.
3. Comparison of PSM results with RCT outcomes across several metrics, including effect size, covariate balance, and receiver operating characteristic curves.
4. Meta-analysis using random-effects modeling to assess differences in effect size estimates.
    

Outcome

Key Findings:

• All seven PSM techniques showed strong correlations with RCT results (r > 0.90).
• Effect sizes (Cohen’s d) from PSM methods were generally comparable to those from RCTs, with differences ranging from 0.03 to 0.09.
• 1-1 matching and optimal pairs matching performed the best in replicating RCT results.
• Some PSM methods tended to overestimate treatment effects compared to RCTs.
   

Implications or Recommendations: 

• PSM is a viable alternative to RCTs for causal inference in criminal justice research when RCTs are infeasible.
• Researchers should use multiple PSM techniques and robust balance measures to ensure reliability.
• Policy recommendations based on PSM should be approached with caution, given the potential for overestimation and situational variability.
• Further research is needed to refine PSM techniques and explore situational factors that affect their performance.

This study offers a methodological validation of PSM in criminal justice research, highlighting its strengths and limitations while providing actionable guidance for researchers and policymakers.