Measuring t-statistics and p-values using Python

Let's fabricate some experimental data and use the t-statistic and p-value to determine whether a given experimental result is a real effect or not. We're going to actually fabricate some fake experimental data and run t-statistics and p-values on them, and see how it works and how to compute it in Python.

Running A/B test on some experimental data

Let's imagine that we're running an A/B test on a website and we have randomly assigned our users into two groups, group A and group B. The A group is going to be our test subjects, our treatment group, and group B will be our control, basically the way the website used to be. We'll set this up with the following code:

import numpy as np 
from scipy import stats 
 
A = np.random.normal(25.0, 5.0, 10000) 
B = np.random.normal(26.0, 5.0, 10000) 
 
stats.ttest_ind(A, B) 

In this code example, our treatment group (A) is going to have a randomly distributed purchase behavior where they spend, on average...