Confidence intervals in Python
Confidence intervals in Python
Setup
# Import libraries
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
sns.set(rc={'figure.figsize':(9,6)})
Generate population data
# Generate normal distributed data
np.random.seed(222)
df = pd.DataFrame({'value': np.random.normal(loc=40, scale=10, size=10000)})
# Plot histogram and mean (in green)
sns.distplot(df);
plt.xlim(10, 70)
plt.axvline(df.mean()[0], color='darkslateblue');
# Check actual mean and standard deviation
df.describe().head(3)
| value | |
|---|---|
| count | 10000.000000 |
| mean | 40.015843 |
| std | 9.936291 |
Generate sample
# Random sample
sample_df = pd.DataFrame(np.random.choice(df['value'], 50))
# Plot sample distribution
sample_df.describe().head(3)
| 0 | |
|---|---|
| count | 50.000000 |
| mean | 41.321748 |
| std | 9.948352 |
Calculate confidence interval
The confidence interval formula is: \(\bar{x}\pm z*\frac{\sigma}{\sqrt{n}}\)
where:
- \(\bar{x}\) is the sample mean
- \(z\) is the z-score
- \(\sigma\) is the sample standard deviation
- \(n\) is the sample size
# Calculate confidence interval
x_bar = sample_df.mean()[0] # Sample mean
sigma = sample_df.std()[0] # Standard deviation
n = len(sample_df) # Sample size
z_score = 1.96 # Z-score for 95% two-sided interval
ci = [x_bar - z_score * sigma / np.sqrt(n), x_bar + z_score * sigma / np.sqrt(n)]
ci
[38.564205758105565, 44.07929074221945]
# Plot histogram and true population mean
sns.distplot(df, hist=False);
plt.xlim(10, 70)
plt.axvline(df.mean()[0], color='darkslateblue')
# Plot sample mean and confidence interval
plt.axvline(x_bar, color='orange', linestyle='--')
plt.axvspan(ci[0], ci[1], alpha=0.3, color='gold');
The plot above shows the population actual distribution with a mean of 40.01, as well as the sample mean (41.32) and a 95% confidence interval of [38.56, 44.08].