Statistical Tests (T-test)

T-tests: A statistical method to determine if there is a significant difference between the means of 2 samples (independent samples or same sample at different conditions/interval)

It calculates a t-statistic, which is the ratio of the difference between group means to the variability in the data. This t-statistic is then compared to critical values from the t-distribution to determine if the difference is statistically significant.

One sample t-test, two sample t-test, paired t-test.

One Sample t-test: The one-sample t-test is often used to compare a sample mean to a claimed or known population value to determine if the sample differs significantly from this benchmark

Example: Testing a Dog Food’s Weight Gain Claim

Suppose a dog food manufacturer claims their product increases a dog’s weight by 3 pounds per month. Uncle Roger tests this claim with his dog.

Scenario 1: If Uncle Roger found it was indeed true, and no other factors were involved, other than food, then the test he did was highly reliable and supports manufacturer claims

Scenario 2: If Uncle Roger found the dog gained 3 lbs, and also claims that he added additional food to the dog food, the results of the test match the claim, but are less reliable because of additional factors.

Scenario 3: If uncle roger found his dog has not gained 3 lbs, after feeding the food, and isolated the dog from other factors, his test is reliable, then the claims made by producers are false (it can’t be confirmed by just uncle rogers test, it should be backed by other people with same conditions).

Scenario 4: If Uncle Roger found his dog had not gained 3 lbs, after feeding the food, and was not isolated from other factors(exercised more, was sick, etc), the test has low reliability because uncontrolled factors could explain the test result.

We could confidently confirm results from scenarios 1 and 3, but that is not the case with scenarios 2 and 4 (due to poor control over other variables).

All these are possible scenarios and should be considered when calculating the test statistic. An actual statistical result would narrow down to determining weight gain (significantly less than 3 lbs [left-tailed test] or greater than 3 lbs [right-tailed test] or different from 3 lbs [2 tailed test]).

Two-Sample t-test: Is often used to compare the difference in the means of 2 independent groups (between subjects of different groups). (2 subjects with 2 interventions)

Example: Producer of Dog Food’s (Food A) Weight Gain claim over (Food B)

Uncle Roger has 2 dogs, and feeds dog A with Food A, dog B with Food B, and checks if the food is making any difference in the weights of the dogs by the end of the month, if dog A gained more weight compared to dog B, assuming other factors were controlled properly, it is clear that Food A appreas to be associated with greater weight gain than Food B.

Paired t-test: Is often used to compare the difference in the means within the group (within the same group of subjects) at different time intervals or under various conditions. If it’s a clinical study, participants would remain the same across measurements. (1 subject with 2 interventions)

Example: Testing a Dog Food’s Weight Gain Claim

Uncle Roger wants to determine whether a particular dog food helps his dog grow. To test this, he measures the dog’s weight before introducing the food and again after feeding it for one month. If there is a noticeable weight change, he concludes that the dog food has had a statistically significant effect. However, you might question whether the weight change is solely due to the dog food or influenced by other factors, such as exercise, etc. If Uncle Roger confirms that no changes were made apart from feeding the dog, the test results are more reliable. However, if he also introduced exercise or other variables, it becomes harder to attribute the weight change exclusively to the dog food. In such cases, a paired t-test may not be as reliable, and alternative methods like ANCOVA or multivariate analysis should be considered.

Limitations of t-tests: Data should satisfy normality and equal variances (for 2-sample t-test). Most reliable for small to moderate datasets. It is limited to continuous data and can’t be used for categorical data.