difference_test()¶

Description¶

Conducts a few different statistical tests which test for a difference between independent or related samples with or without equal variances and has the ability to calculate the effect size of the observed difference. The data is returned in a Pandas DataFrame by default, but can be returned as a dictionary if specified.

This method is similar to researchpy.ttest(), except it allows the user to use the formula syntax.

This method can perform the following tests:

Independent sample t-test 1

psudo-code: difference_test(formula_like, data, equal_variances = True, independent_samples = True)

Paired sample t-test 2

psudo-code: difference_test(formula_like, data, equal_variances = True, independent_samples = False)

Welch’s t-test 1

psudo-code: difference_test(formula_like, data, equal_variances = False, independent_samples = True)

Wilcoxon signed-rank test 3

By default, discards all zero-differences; this is known as the ‘wilcox’ method.

psudo-code: difference_test(formula_like, data, equal_variances = False, independent_samples = False)

2 objects will be returned for all available tests except for the Wilcoxon signed-rank test; the first object will be a descriptive summary table and the second will be the testing result information which will include the effect size measures if indicated.

For the Wilcoxon signed-rank test, 3 objects will be returned. The first object provides descriptive information regarding the ranks, the second object contains the adjustment information, and the third object contains the test results.

Parameters¶

Input¶

difference_test(formula_like, data = {}, conf_level = 0.95, equal_variances = True, independent_samples = True, wilcox_parameters = {“zero_method” : “wilcox”, “correction” : False, “mode” : “auto”}, welch_dof = “satterthwaite”, **keywords)

formula_like: A valid formula ; for example, “DV ~ IV”.

data: data to perform the analysis on - contains the dependent and independent variables.

conf_level: Specify the confidence interval to be calculated.

equal_variances: Boolean to indicate if equal variances are assumed.

independent_samples: Boolean to indicate if groups and independent of each other.

wilcox_parameters: A dictionary with optional methods for calculating the Wilcoxon signed-rank test. For more information, see scipy.stats.wilcoxon.

welch_dof : A string which indicates how to calculate the degrees of freedom for the Welch’s t-test; options are “satterthwaite” (default) and “welch”.

Returns¶

Returns an object with class “difference_test”; this object has accessible methods which are described below.

difference_test methods¶

conduct(return_type = “Dataframe”, effect_size = None, decimals = 4)
- return_type: Specify if the results should be returned as a Pandas DataFrame (default) or a Python dictionary (= ‘Dictionary’).
- effect_size: Specify if effect sizes should be calculated, default value is None. * Available options are: None, “Cohen’s D”, “Hedge’s G”, “Glass’s delta1”, “Glass’s delta2”, “r”, and/or “all”. * User can specify any combination of effect sizes, or use “all” which will calculate all effect sizes. * Only effect size “r” is supported for the Wilcoxon ranked-sign test.
- decimals : The number of decimal places the data should be rounded too.
note

If

returns

2 objects will be returned within a tuple:

First object will be the descriptive summary information.

Second object will be the statistical testing information.

Note

This can be a one step, or two step process.

One step .. code:: python

difference_test(“DV ~ IV”, data).conduct()

Two step .. code:: python

model = difference_test(“DV ~ IV”, data) model.conduct()

Welch Degrees of freedom¶

There are two degrees of freedom options available when calculating the Welch’s t-test. The default is to use the Satterthwaite 4 calculation with the option to use the Welch 5 calculation.

Satterthwaite (1946)¶

\[\frac{(\frac{s^2_x}{n_x} + \frac{s^2_y}{n_y})^2}{\frac{(\frac{s^2_x}{n_x})^2}{n_x-1} + \frac{(\frac{s^2_y}{n_y})^2}{n_y-1} }\]

Welch (1947)¶

\[-2 + \frac{(\frac{s^2_x}{n_x} + \frac{s^2_y}{n_y})^2}{\frac{(\frac{s^2_x}{n_x})^2}{n_x+1} + \frac{(\frac{s^2_y}{n_y})^2}{n_y+1}}\]

Effect Size Measures Formulas¶

Cohen’s d_s (between subjects design)¶

Cohen’s d_s 6 for a between groups design is calculated with the following equation:

\[d_s = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\frac{(n_1 - 1)SD^2_1 + (n_2 - 1)SD^2_2}{n_1 + n_2 - 2}}}\]

Cohen’s d_av (within subject design)¶

Another version of Cohen’s d is used in within subject designs. This is noted by the subscript “av”. The formula for Cohen’s d_av 7 is as follows:

\[d_{av} = \frac{M_{diff}}{\frac{SD_{1} + SD_{2}}{2}}\]

Hedges’s g_s (between subjects design)¶

Cohen’s d_s gives a biased estimate of the effect size for a population and Hedges and Olkin 8 provides an unbiased estimation. The differences between Hedges’s g and Cohen’s d is negligible when sample sizes are above 20, but it is still preferable to report Hedges’s g 9. Hedge’s g_s is calculated using the following formula:

\[\text{Hedges's g}_s = \text{Cohen's d}_s \times (1 - \frac{3}{4(n_1 + n_2 - 9)})\]

Hedges’s g_av (within subjects design)¶

Cohen’s d_av gives a biased estimate of the effect size for a population and Hedges and Olkin 8 provides a correction to be applied to provide an unbiased estimate. Hedge’s g_av is calculated using the following formula 7 :

\[\text{Hedges's g}_{av} = \text{Cohen's d}_av \times (1 - \frac{3}{4(n_1 + n_2 - 9)})\]

Glass’s \(\Delta\) (between or within subjects design)¶

Glass’s \(\Delta\) is the mean differences between the two groups divided by the standard deviation of the first condition/group or by the second condition/group. When used in a within subjects design, it is recommended to use the pre- standard deviation in the denominator 7; the following formulas are used to calculate Glass’s \(\Delta\):

\[ \begin{align}\begin{aligned}\Delta_1 = \frac{(\bar{x}_1 - \bar{x}_2)}{SD_1}\\\Delta_2 = \frac{(\bar{x}_1 - \bar{x}_2)}{SD_2}\end{aligned}\end{align} \]

Pearson correlation coefficient r (between or within subjects design)¶

Rosenthal 10 provided the following formula to calculate the Pearson correlation coefficient r using the t-value and degrees of freedom:

\[r = \frac{t}{\sqrt{t^2 + df}}\]

Rosenthal 10 provided the following formula to calculate the Pearson correlation coefficient r using the z-value and N. This formula is used to calculate the r coefficient for the Wilcoxon ranked-sign test. Note, that N is the total number of observations.

\[r = \frac{Z}{\sqrt{N}}\]

Rank-Biserial correlation coefficient r (between or within subjects design)¶

The following formula is used to calculate the Rank-Biserial correlation coefficient r 11 for the Wilcoxon ranked-sign test.

\[\text{Rank-Biserial r = } \frac{\sum{Ranks}_{+} - \sum{Ranks}_{-}}{\sum{Ranks}_{total}}\]

Examples¶

Loading Packages and Data¶

First let’s create an example data set to work through the examples. This will be done using numpy (to create fake data) and pandas (to hold the data in a data frame).

import numpy, pandas, researchpy

numpy.random.seed(12345678)

df = pandas.DataFrame(numpy.random.randint(10, size= (100, 2)),
                  columns= ['No', 'Yes'])

df["id"] = range(1, df.shape[0] + 1)

df.head()

No  Yes  id
  2   1
  1   2
  1   3
  2   4
  6   5

If one has data like this and doesn’t want to reshape the data, then researchpy.different_test() will not work and one should use researchpy.ttest() instead. However, moving forward researchpy will be going in the direction of syntax style input and it is recommended to get comfortable using this approach if one plans to use researchpy in the future.

Currently the data is in a wide format and it needs to be in a long format, i.e. one variable with the dependent variable data and another with the independent variable data. The current data structure won’t work and it needs to be reshaped; there are a few ways to do this, one will be shown below.

df2 = pandas.melt(df, id_vars = "id", value_vars = ["No", "Yes"],
                  var_name = "Exercise", value_name = "StressReactivity")

df2.head()

id Exercise  StressReactivity
     No                 3
     No                 4
     No                 0
     No                 8
     No                 6

Now the data is in the correct structure.

Independent Samples t-test¶

# Independent t-test

# If you don't store the 2 returned DataFrames, it outputs as a tuple and
# is displayed
rp.difference_test("StressReactivity ~ C(Exercise)",
                   data = df2,
                   equal_variances = True,
                   independent_samples = True).conduct(effect_size = "all")

(      Variable      N   Mean        SD        SE  95% Conf.  Interval
    healthy  100.0  4.590  2.749086  0.274909   4.044522  5.135478
non-healthy  100.0  4.160  3.132495  0.313250   3.538445  4.781555
   combined  200.0  4.375  2.947510  0.208420   3.964004  4.785996,
                                  Independent t-test   results
           Difference (healthy - non-healthy) =     0.4300
                           Degrees of freedom =   198.0000
                                            t =     1.0317
                        Two side test p value =     0.3035
                       Difference < 0 p value =     0.8483
                       Difference > 0 p value =     0.1517
                                    Cohen's d =     0.1459
                                    Hedge's g =     0.1454
                                Glass's delta =     0.1564
                                            r =     0.0731)

# Otherwise you can store them as objects
summary, results = rp.difference_test("StressReactivity ~ C(Exercise)",
                                      data = df2,
                                      equal_variances = True,
                                      independent_samples = True).conduct(effect_size = "all")

summary

       Name    N   Mean Variance       SD        SE  95% Conf.  Interval
      No  100  4.590  7.55747  2.74909  0.274909   4.044522  5.135478
     Yes  100  4.160  9.81253   3.1325  0.313250   3.538445  4.781555
combined  200  4.375  8.68781  2.94751  0.208420   3.964004  4.785996
    diff       0.430                    0.416773  -0.391884  1.251884

results

   Independent samples t-test     Results
     Difference (No - Yes)    0.430000
      Degrees of freedom =  198.000000
                       t =    1.031736
  Two sided test p-value =    0.303454
  Difference < 0 p-value =    0.848273
  Difference > 0 p-value =    0.151727
                Cohen's Ds    0.145909
                 Hedge's G    0.145356
            Glass's delta1    0.156416
            Glass's delta2    0.137271
         Point-Biserial r    0.073126

Paired Samples t-test¶

# Paired samples t-test
summary, results = rp.difference_test("StressReactivity ~ C(Exercise)",
                                      data = df2,
                                      equal_variances = True,
                                      independent_samples = False).conduct(effect_size = "all")

summary

   Name    N  Mean Variance        SD        SE  95% Conf.  Interval
  No  100  4.59  7.55747  2.749086  0.274909   4.044522  5.135478
 Yes  100  4.16  9.81253  3.132495  0.313250   3.538445  4.781555
diff       0.43           4.063275  0.406327  -0.376242  1.236242

results

       Paired samples t-test    Results
    Difference (No - Yes)   0.430000
     Degrees of freedom =  99.000000
                      t =   1.058260
 Two sided test p-value =   0.292512
 Difference < 0 p-value =   0.853744
 Difference > 0 p-value =   0.146256
              Cohen's Dav   0.146219
              Hedge's Gav   0.145665
           Glass's delta1   0.156416
           Glass's delta2   0.137271
        Point-Biserial r   0.105763

Welch’s t-test¶

One can request either the Satterthwaite (default) or Welch degrees of freedom; to calculate degrees of freedom using Welch’s formula set welch_dof = “welch”

# Welch's t-test
summary, results = rp.difference_test("StressReactivity ~ C(Exercise)",
                                   data = df2,
                                   equal_variances = False,
                                   independent_samples = True).conduct(effect_size = "all")

summary

       Name    N   Mean Variance       SD        SE  95% Conf.  Interval
      No  100  4.590  7.55747  2.74909  0.274909   4.044522  5.135478
     Yes  100  4.160  9.81253   3.1325  0.313250   3.538445  4.781555
combined  200  4.375  8.68781  2.94751  0.208420   3.964004  4.785996
    diff       0.430                    0.416773  -0.391919  1.251919

results

              Welch's t-test     Results
    Difference (No - Yes)    0.430000
     Degrees of freedom =  196.651845
                      t =    1.031736
 Two sided test p-value =    0.303476
 Difference < 0 p-value =    0.848268
 Difference > 0 p-value =    0.151732
               Cohen's Ds    0.145909
                Hedge's G    0.145356
           Glass's delta1    0.156416
           Glass's delta2    0.137271
        Point-Biserial r    0.073375

Wilcoxon signed-rank Test¶

# Wilcoxon signed-rank test
desc, var_adj, res = difference_test("StressReactivity ~ C(Exercise)", df2, independent_samples=False, equal_variances = False).conduct()

print(desc, var_adj, res, sep = "\n"*2)

sign	obs	sum ranks	expected
positive	40	2,298.0000	2,502.5000
negative	51	2,707.0000	2,502.5000
zero	9	45.0000	45.0000
all	100	5,050.0000	5,050.0000

unadjusted variance	adjustment for ties	adjustment for zeros	adjusted variance
84,587.5000	-375.1250	-71.2500	84,141.1250

z	w	pval
-0.7050	2,298.0000	0.4808

References¶

1(1,2): scipy.stats.ttest_ind. The SciPy community, 2018. Retrieved when last updated on May 5, 2018. URL: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html.
2: scipy.stats.ttest_rel. The SciPy community, 2018. Retrieved when last updated on May 5, 2018. URL: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.html.
3: scipy.stats.wilcoxon. The SciPy community, 2018. Retrieved when last updated on May 5, 2018. URL: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html.
4: F. E. Satterthwaite. An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6):110–114, 1946. URL: http://www.jstor.org/stable/3002019.
5: B. L. Welch. The generalization of `student’s’ problem when several different population variances are involved. Biometrika, 34(1/2):28–35, 1947. URL: http://www.jstor.org/stable/2332510.
6: Jacob Cohen. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, second edition, 1988. ISBN 0-8058-0283-5.
7(1,2,3): Daniel Lakens. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and anovas. Frontiers in Psychology, November 2013. doi:10.3389/fpsyg.2013.00863.
8(1,2): Larry Hedges and Ingram Olkin. Journal of Educational Statistics, chapter Statistical Methods in Meta-Analysis. Volume 20. Academic Press, Inc., 1985, 10.2307/1164953.
9: Rex B. Kline. Beyond significance testing: Reforming data analysis methods in behavioral research. American Psychological Association, 2004. http://dx.doi.org/10.1037/10693-000.
10(1,2): Robert Rosenthal. The hand-book of research synthesis, chapter Parametric measures of effect size, pages 231–244. New York, NY: Russel Sage Foundation, 1994.
11: Dave S. Kerby. The simple difference formula: an approach to teaching nonparametric correlation. Innovative Teaching, 2014. 10.2466/11.IT.3.1.