Test Selection Guide

# Statistical Test Selection Guide This guide provides a decision tree for selecting appropriate statistical tests based on research questions, data types, and study designs. ## Decision Tree for Test Selection ### 1. Comparing Groups #### Two Independent Groups - **Continuous outcome, normally distributed**: Independent samples t-test - **Continuous outcome, non-normal**: Mann-Whitney U test (Wilcoxon rank-sum test) - **Binary outcome**: Chi-square test or Fisher's exact test (if expected counts 2 raters**: Fleiss' kappa or Krippendorff's alpha - **Continuous ratings**: Intraclass correlation coefficient (ICC) #### Test-Retest Reliability - **Continuous measurements**: ICC or Pearson correlation - **Internal consistency**: Cronbach's alpha #### Agreement Between Methods - **Continuous measurements**: Bland-Altman analysis - **Categorical classifications**: Cohen's kappa ### 4. Categorical Data Analysis #### Contingency Tables - **2x2 table**: Chi-square test or Fisher's exact test - **Larger than 2x2**: Chi-square test - **Ordered categories**: Cochran-Armitage trend test - **Paired categories**: McNemar's test (2x2) or McNemar-Bowker test (larger) ### 5. Bayesian Alternatives Any of the above tests can be performed using Bayesian methods: - **Group comparisons**: Bayesian t-test, Bayesian ANOVA - **Correlations**: Bayesian correlation - **Regression**: Bayesian linear/logistic regression **Advantages of Bayesian approaches:** - Provides probability of hypotheses given data - Naturally incorporates prior information - Provides credible intervals instead of confidence intervals - No p-value interpretation issues ## Key Considerations ### Sample Size - Small samples (n < 30): Consider non-parametric tests or exact methods - Very large samples: Even small effects may be statistically significant; focus on effect sizes ### Multiple Comparisons - When conducting multiple tests, adjust for multiple comparisons using: - Bonferroni correction (conservative) - Holm-Bonferroni (less conservative) - False Discovery Rate (FDR) control (Benjamini-Hochberg) - Tukey HSD for post-hoc ANOVA comparisons ### Missing Data - Complete case analysis (listwise deletion) - Multiple imputation - Maximum likelihood methods - Ensure missing data mechanism is understood (MCAR, MAR, MNAR) ### Effect Sizes - Always report effect sizes alongside p-values - See `effect_sizes_and_power.md` for guidance ### Study Design Considerations - Randomized controlled trials: Standard parametric/non-parametric tests - Observational studies: Consider confounding and use regression/matching - Clustered/nested data: Use mixed-effects models or GEE - Time series: Use time series methods (ARIMA, etc.)