Confidence intervals and hypothesis tests

#Why it matters

"Statistical analysis" and "hypothesis testing" appear in analyst and data scientist job descriptions because businesses constantly compare groups: this campaign against that one, this city against another. A difference in a table is not automatically a real difference. Interviewers test whether you know how to tell.

#The concept

• A sample is the data you have. The population is what you want to know about.

• A confidence interval gives a range of plausible values for a population number. A 95% interval is built by a method that captures the true value in 95% of repeated samples.

• A hypothesis test starts from a null hypothesis (no difference) and asks how surprising your data would be if it were true. The p-value is that surprise: the probability of a difference at least this large if there were really none.

• A small p-value (commonly below 0.05) says the data would be unusual under the null. It does not say the effect is large, important or caused by what you think.

#Worked example: does the contact channel matter?

The UCI Bank Marketing dataset (CC BY 4.0) records 45,211 phone calls by a Portuguese bank selling term deposits. 5,289 calls ended in a subscription.

import pandas as pd
from statsmodels.stats.proportion import proportions_ztest, proportion_confint

bank = pd.read_csv("bank-full.csv", sep=";")
summary = bank.groupby("contact")["y"].agg(
    calls="count", subscribed=lambda s: (s == "yes").sum())
summary["rate"] = summary.subscribed / summary.calls
print(summary)

cell, tel = summary.loc["cellular"], summary.loc["telephone"]
z, p = proportions_ztest([cell.subscribed, tel.subscribed], [cell.calls, tel.calls])
print(f"z = {z:.2f}, p = {p:.3f}")
for name, row in [("cellular", cell), ("telephone", tel)]:
    low, high = proportion_confint(row.subscribed, row.calls, method="wilson")
    print(name, f"{row.rate:.1%} (95% CI {low:.1%} to {high:.1%})")

The test gives z = 2.17 and p = 0.030. Under the usual 0.05 threshold, the difference is statistically significant.

Now think before you recommend "call mobiles":

• The telephone interval is wide because there are ten times fewer landline calls.

• The groups differ in other ways. Landline customers are older on average (47.8 years against 40.4). Age, not the channel, may explain the gap.

• The bank chose who to call and how. This is observational data, not an experiment, so the test tells you the difference is unlikely to be noise; it cannot tell you the channel caused it.

#Common mistakes

• "p = 0.03 means a 97% chance the effect is real." It does not. It is the probability of data this extreme if there were no effect.

• Ignoring effect size. With millions of rows, tiny meaningless differences become significant. Always report the difference and its interval.

• Testing many things and reporting the one that worked. Twenty tests at the 0.05 level will produce about one false positive by chance.

• Treating observational differences as causal. Only randomised experiments, or careful causal methods, support "X caused Y".

#Interview angle

"Explain a p-value to a non-technical stakeholder" is extremely common. Try: "If the channel made no difference, we would see a gap this big only about 3 times in 100. So the gap is probably real, but it may come from who we called, not how."

#Practice task

1. Compare subscription rates for customers whose previous campaign outcome (poutcome) was success versus failure. Report rates, intervals and a p-value.

2. Repeat the contact test only for customers aged 30 to 45. Does the gap survive?

3. Write three sentences for a marketing manager.

#Further reading

• UCI Bank Marketing dataset (CC BY 4.0)

• statsmodels: proportion tests

• The ASA statement on p-values