Variance and standard deviation explained

#Why it matters

Standard deviation explained in one line: the typical distance of a value from the average. Every analyst uses it, often without noticing which version they computed. Two customers can wait the same 12 days on average, and one of them gets the parcel on day 11 every time while the other waits anywhere from 3 to 40 days. Only a spread measure tells them apart, and in logistics, payments and SLAs that spread is the thing customers complain about. Statistics sits in 58% of Data Analyst and 82% of Data Scientist job descriptions we read, and "standard deviation vs standard error" is a staple screening question.

#The concept

#Variance and standard deviation

The variance is the average squared distance from the mean. For a population of $N$ values:

Squaring does two jobs: it makes every distance positive, and it punishes large distances much more than small ones. The cost is units: if delivery time is in days, variance is in "days squared", which nobody can picture. The standard deviation is the square root of the variance, $\sigma =\sqrt{{\sigma }^2}$, and it is back in days. Report the SD; use the variance in formulas (variances of independent things add; SDs do not).

#Population (n) or sample (n - 1)

When your data is a sample and you want to estimate the spread of the population it came from, divide by $n-1$ instead of $n$:

This is Bessel's correction. The reason: you measure distances from the sample mean $\bar{x}$, which is by construction the point closest to your sample. Distances to the true mean $\mu$ would be a little larger on average. Dividing by $n-1$ inflates the result by exactly enough to make the variance estimate unbiased. You "used up" one degree of freedom to estimate the mean.

In practice: with 41 values the two differ by about 1.2%; with 40,000 values the difference is invisible. It matters for small samples, and it matters for matching someone else's number.

#Different tools, different defaults

ddof means "delta degrees of freedom": the divisor is n - ddof. The classic bug is a pandas DataFrame column converted with .to_numpy() and then .std(): the number changes and nobody knows why.

#Coefficient of variation

An SD of 7 days means something different for a delivery that averages 9 days and one that averages 29. The coefficient of variation divides the SD by the mean, $CV=s/\bar{x}$, giving spread relative to the level. Use it to compare spread across groups or units with different scales (rupee prices and dollar prices, small and large stores). It only makes sense for ratio data with a true zero, and it explodes when the mean is near zero.

#The 68-95-99.7 rule, and when it lies

For a normal distribution, about 68% of values fall within 1 SD of the mean, 95% within 2 and 99.7% within 3. People quote this for any data. On skewed data it is wrong: you will see below that delivery times have 84% within 1 SD, not 68%, because a long right tail inflates the SD while most values cluster near the middle. Check the shape before using the rule.

#Standard deviation vs standard error

• The standard deviation describes the spread of individual values: how different one delivery is from another.

• The standard error of the mean, $SE=s/\sqrt{n}$, describes how much the sample mean would move if you repeated the sampling. It measures the precision of an estimate.

More data does not shrink the SD (deliveries do not become more consistent because you collected more of them). More data does shrink the SE. The central limit theorem lesson in the next module shows why by simulation; for now remember: SD for describing, SE for uncertainty.

#MAD: a spread measure that ignores the tail

Because it squares distances, the SD is dominated by the most extreme values. The median absolute deviation is the median of $|x_i-\text{median}(x)|$. Multiplied by 1.4826 it equals the SD for normal data, so it is directly comparable; SciPy does this with scale="normal". Together with the IQR it is the robust choice for skewed or dirty data.

#Worked example: how consistent are deliveries in each Brazilian state?

The Olist dataset is around 100,000 real orders from a Brazilian marketplace between 2016 and 2018, published by Olist on Kaggle under CC BY-NC-SA 4.0 and mirrored in Olist's GitHub repository. You need the orders and customers tables.

#Step 1: delivery days per order

import numpy as np
import pandas as pd
from scipy import stats

base = "https://raw.githubusercontent.com/olist/work-at-olist-data/master/datasets/"
orders = pd.read_csv(base + "olist_orders_dataset.csv",
                     parse_dates=["order_purchase_timestamp", "order_delivered_customer_date",
                                  "order_estimated_delivery_date"])
customers = pd.read_csv(base + "olist_customers_dataset.csv")

df = orders.merge(customers, on="customer_id")
df = df[df["order_status"] == "delivered"].dropna(subset=["order_delivered_customer_date"])
df["days"] = (df["order_delivered_customer_date"]
              - df["order_purchase_timestamp"]).dt.total_seconds() / 86400
print(f"n = {len(df):,}")
print(df["days"].describe().round(2))

n = 96,470
count    96470.00
mean        12.56
std          9.55
min          0.53
25%          6.77
50%         10.22
75%         15.72
max        209.63
Name: days, dtype: float64

#Step 2: the ddof trap

days = df["days"]
arr = days.to_numpy()
print("pandas  .std()        ", days.std())
print("numpy   .std()        ", arr.std())
print("numpy   .std(ddof=1)  ", arr.std(ddof=1))

rr = df.loc[df["customer_state"] == "RR", "days"]
print(f"Roraima, n = {len(rr)}: ddof=0 -> {rr.std(ddof=0):.3f}, ddof=1 -> {rr.std(ddof=1):.3f}")

pandas  .std()         9.54615637792742
numpy   .std()         9.546106900467501
numpy   .std(ddof=1)   9.54615637792742
Roraima, n = 41: ddof=0 -> 27.402, ddof=1 -> 27.742

On 96,470 orders the two versions agree to four decimal places, so nobody notices. On Roraima's 41 orders they differ by about 0.34 days, enough to fail a reconciliation with someone's Excel sheet. Pick one convention for sample data (pandas' ddof=1, Excel's STDEV.S) and use it everywhere.

#Step 3: does the 68-95-99.7 rule hold?

m, s = days.mean(), days.std()
for k in (1, 2, 3):
    share = ((days - m).abs() <= k * s).mean()
    print(f"within {k} SD: {share:.1%}")

within 1 SD: 83.8%
within 2 SD: 96.0%
within 3 SD: 98.4%

84% within 1 SD instead of 68%: delivery time is right-skewed (median 10.2 days, mean 12.6, maximum 210), so the SD is inflated by the long tail and the "68%" band is too wide for the bulk of orders. Never quote the rule for skewed data.

#Step 4: spread per state, four ways

g = df.groupby("customer_state")["days"]
table = pd.DataFrame({
    "n": g.size(),
    "mean": g.mean(),
    "median": g.median(),
    "sd": g.std(),
    "iqr": g.quantile(0.75) - g.quantile(0.25),
    "mad": g.apply(lambda x: stats.median_abs_deviation(x, scale="normal")),
})
table["cv"] = table["sd"] / table["mean"]
table["se_mean"] = table["sd"] / np.sqrt(table["n"])
late = df["order_delivered_customer_date"].dt.normalize() > df["order_estimated_delivery_date"]
table["late_share"] = late.groupby(df["customer_state"]).mean()
print(table.sort_values("mean").round(2).to_string())

                    n   mean  median     sd    iqr    mad    cv  se_mean  late_share
customer_state                                                                      
SP              40494   8.76    7.21   6.75   6.11   4.35  0.77     0.03        0.04
PR               4923  11.99   10.43   6.98   7.11   4.96  0.58     0.10        0.04
MG              11354  12.01   10.31   7.20   7.15   4.90  0.60     0.07        0.05
DF               2080  12.97   11.36   7.06   7.99   5.45  0.54     0.15        0.06
SC               3546  14.95   13.01   8.58   9.77   6.95  0.57     0.14        0.08
RS               5344  15.30   13.18   9.18   9.12   6.24  0.60     0.13        0.06
RJ              12350  15.31   12.04  11.52  10.36   6.98  0.75     0.10        0.12
GO               1957  15.61   13.95   9.28   8.22   5.89  0.59     0.21        0.07
MS                701  15.62   13.94   7.73   8.30   6.00  0.49     0.29        0.10
ES               1995  15.79   13.64  10.64   8.96   6.54  0.67     0.24        0.11
TO                274  17.66   15.87   7.58   8.69   6.48  0.43     0.46        0.10
MT                886  18.06   16.11   8.50   8.43   6.03  0.47     0.29        0.06
PE               1593  18.45   15.70  10.95  10.68   7.72  0.59     0.27        0.10
RN                474  19.28   16.06  12.99  10.96   7.57  0.67     0.60        0.09
BA               3256  19.34   16.91  11.69  10.72   7.46  0.60     0.20        0.12
RO                243  19.37   17.58   7.69   8.70   6.41  0.40     0.49        0.03
PI                476  19.46   16.29  14.72  10.12   7.55  0.76     0.67        0.14
PB                517  20.43   18.15  10.92  11.99   8.57  0.53     0.48        0.10
AC                 80  21.04   18.36  10.83  10.18   6.85  0.51     1.21        0.04
CE               1279  21.27   18.21  13.13  11.45   7.81  0.62     0.37        0.14
SE                335  21.52   18.01  16.76  11.19   7.68  0.78     0.92        0.15
MA                717  21.57   19.19  11.35  10.25   7.46  0.53     0.42        0.17
PA                946  23.77   21.08  13.15  13.67   9.73  0.55     0.43        0.11
AL                397  24.54   22.33  11.48  14.55  10.52  0.47     0.58        0.21
AM                145  26.43   25.88  13.86  12.93   9.78  0.52     1.15        0.03
AP                 67  27.19   24.35  21.38  11.03   8.35  0.79     2.61        0.03
RR                 41  29.39   25.01  27.74  18.52  14.58  0.94     4.33        0.12

What the table tells you:

• São Paulo (SP) is fastest on average (8.8 days) but not the most consistent in relative terms: CV 0.77, among the highest. Its SD is small in days, but large compared to its mean.

• Rio de Janeiro (RJ) and Goiás (GO) have almost the same mean (15.3 vs 15.6 days), yet RJ's SD is 11.5 against GO's 9.3. RJ's MAD (7.0) and IQR (10.4) are less dramatic: RJ's extra SD comes from a long tail of very slow orders, the ones that turn into support tickets.

• Late share follows spread, not the mean. Amazonas (AM) has a 26-day average but only about 3% of orders arrive after the promised date, because Olist quotes long estimates there. RJ is about 12% late on a 15-day average. Customers judge you against the promise, and you can only promise what your spread allows.

• Small states are noisy. Roraima (RR) has the highest SD (27.7) on just 41 orders, and its standard error of the mean is 4.3 days against 0.03 for SP. You cannot rank RR confidently against its neighbours.

#Step 5: rank states by consistency, then test it against promises

Rank the large states (at least 500 orders, so the ranking is not noise) by IQR in days, and put the delivery promise next to it. Olist quotes an estimated delivery date at purchase; the median promised days per state is how much room the promise leaves.

promised = (df["order_estimated_delivery_date"] - df["order_purchase_timestamp"]).dt.days
table["promised_days"] = promised.groupby(df["customer_state"]).median()
table["p90"] = g.quantile(0.9)
table["cushion"] = table["promised_days"] - table["p90"]

big = table[table["n"] >= 500].sort_values("iqr")
print(big[["n", "mean", "iqr", "sd", "p90", "promised_days", "cushion", "late_share"]].round(2).to_string())
for col in ["mean", "sd", "iqr", "cushion"]:
    rho = stats.spearmanr(big[col], big["late_share"]).statistic
    print(f"Spearman correlation of {col:>8} with late share: {rho:+.2f}")

                    n   mean    iqr     sd    p90  promised_days  cushion  late_share
customer_state                                                                       
SP              40494   8.76   6.11   6.75  15.77           18.0     2.23        0.04
PR               4923  11.99   7.11   6.98  20.12           24.0     3.88        0.04
MG              11354  12.01   7.15   7.20  20.20           23.0     2.80        0.05
DF               2080  12.97   7.99   7.06  21.64           24.0     2.36        0.06
GO               1957  15.61   8.22   9.28  24.84           26.0     1.16        0.07
MS                701  15.62   8.30   7.73  24.65           25.0     0.35        0.10
MT                886  18.06   8.43   8.50  27.76           31.0     3.24        0.06
ES               1995  15.79   8.96  10.64  25.35           25.0    -0.35        0.11
RS               5344  15.30   9.12   9.18  25.88           28.0     2.12        0.06
SC               3546  14.95   9.77   8.58  25.79           25.0    -0.79        0.08
MA                717  21.57  10.25  11.35  34.32           30.0    -4.32        0.17
RJ              12350  15.31  10.36  11.52  29.09           24.0    -5.09        0.12
PE               1593  18.45  10.68  10.95  30.91           31.0     0.09        0.10
BA               3256  19.34  10.72  11.69  31.81           29.0    -2.81        0.12
CE               1279  21.27  11.45  13.13  34.72           31.0    -3.72        0.14
PB                517  20.43  11.99  10.92  34.95           32.0    -2.95        0.10
PA                946  23.77  13.67  13.15  39.23           36.0    -3.23        0.11
Spearman correlation of     mean with late share: +0.83
Spearman correlation of       sd with late share: +0.90
Spearman correlation of      iqr with late share: +0.81
Spearman correlation of  cushion with late share: -0.93

Read it like a logistics analyst:

• Ranked by IQR, the most consistent large states are São Paulo, Paraná and Minas Gerais, which are also the fastest. Rio de Janeiro has the sixth-largest IQR of these 17 states despite a middling mean (15.3 days).

• Across these 17 states, late share moves more closely with the SD than with the mean. Spread drives broken promises.

• The strongest signal is the cushion: promised days minus the 90th percentile of actual days. Where the promise sits below the 90th percentile (RJ, MA, CE, PA), more than one order in ten arrives late. A promise is a percentile decision, and you can only set it if you measured spread.

#Common mistakes

• Mixing numpy and pandas defaults. np.std divides by n, pd.Series.std by n - 1. Set ddof explicitly in shared code.

• Using the SD to describe skewed data without checking. The SD of delivery days is driven by a tail of 100-plus-day orders. Report the IQR or MAD alongside it.

• Applying 68-95-99.7 to anything. It is a property of the normal distribution, not of data in general. Here 84% of orders sit within 1 SD.

• Comparing raw SDs across very different means. A 7-day SD is tight for a 26-day route and loose for an 8-day one. Use the CV.

• Reporting the SD when you mean the standard error, or the reverse. Error bars on a chart of means should usually be SEs or confidence intervals; a description of how varied customers are should be the SD. Label which one you show.

• Ranking small groups. A group with 41 rows can top any spread ranking by chance. Set a minimum n.

#Interview angle

#Practice task

Using the same Olist tables:

1. Add the olist_order_payments_dataset.csv table and compute mean, SD (ddof=1), IQR, MAD and CV of payment_value per payment_type.

2. Reproduce one of the SDs in Excel or Google Sheets with STDEV.S, and show what STDEV.P gives.

3. For SP only, compute the SD of delivery days per purchase month. Did delivery become more consistent over 2017 to 2018?

4. Write three sentences for a logistics manager: which two large states have the least consistent delivery, how you measured it, and what promise you would set for them (hint: a high percentile, not the mean).

Done when your sentences quote n, a percentile and a consistency measure for each state.

payments = pd.read_csv(base + "olist_order_payments_dataset.csv")
pg = payments.groupby("payment_type")["payment_value"]
pay = pd.DataFrame({"n": pg.size(), "mean": pg.mean(), "sd": pg.std(),
                    "iqr": pg.quantile(.75) - pg.quantile(.25),
                    "mad": pg.apply(lambda x: stats.median_abs_deviation(x, scale="normal"))})
pay["cv"] = pay["sd"] / pay["mean"]
print(pay.round(2))

sp = df[df["customer_state"] == "SP"]
print(sp.groupby(sp["order_purchase_timestamp"].dt.to_period("M"))["days"].agg(["size", "median", "std"]).round(2))
p90 = df.groupby("customer_state")["days"].quantile(0.9)
print(p90.round(1).sort_values())

#Further reading

• NumPy std: the ddof parameter and its default of 0.

• pandas Series.std: default ddof=1.

• SciPy median_abs_deviation: the scale="normal" option.

• NIST/SEMATECH e-Handbook: measures of scale: variance, MAD, IQR and their robustness.