Mean vs median: central tendency and outliers

#Why it matters

Mean vs median is the first statistics decision you make on any new table, and the one most reports get wrong. Statistics appears in 58% of the Data Analyst job descriptions we read and 82% of the Data Scientist ones, and the daily form of that skill is this: pick the summary that tells the truth about the data, say how many rows it covers, and explain what you removed. "When would you report the median instead of the mean?" is one of the most common screening questions for analyst roles.

#The concept

#Mean, median and mode

• The mean adds every value and divides by the count. Every value pulls on it, so one extreme value can drag it a long way.

• The median is the middle value after sorting. Half the values sit below it, half above. Extreme values barely move it.

• The mode is the most common value. It is the only one that works for categories ("most common payment method"), and for prices with round numbers it often lands on a popular price point.

On symmetric data the three agree. On right-skewed data (a long tail of large values, which is what money, time and counts usually look like) the order is mode < median < mean. The tail pulls the mean towards it. On left-skewed data (exam scores out of 100 where most people pass) the order flips.

#Percentiles and the interquartile range

The p-th percentile is the value below which p% of the data falls. The 25th, 50th and 75th percentiles are the quartiles Q1, median and Q3. The interquartile range is $IQR=Q3-Q1$: the width of the middle half of the data. Like the median, it ignores the tails, which makes it the robust partner of the median. Standard deviation is the partner of the mean, and inherits its sensitivity (the next lesson covers it properly).

#The 1.5 IQR rule

John Tukey's rule flags a value as a candidate outlier if it falls below $Q1-1.5\times IQR$ or above $Q3+1.5\times IQR$. It is the rule behind the whiskers of a box plot. Know its limits:

• On skewed data it flags a large share of perfectly real values (you will see 9.2% of taxi trips flagged below).

• It knows nothing about your business. A 300-rupee coffee and a 30,000-rupee laptop can both be normal orders.

• It is a way to find rows to look at, not a rule for deleting them.

#Trimmed means

A trimmed mean drops a fixed share from each end and averages the rest. A 5% trimmed mean ignores the lowest 5% and highest 5%. It sits between the mean (uses everything) and the median (which is the 50% trimmed mean). Olympic diving scores and many inflation measures use the same idea.

#Data types decide which summaries are allowed

Two traps follow from this table. Payment type is stored as the numbers 1 to 6 in the taxi data, so mean(payment_type) runs happily and means nothing. And the average of a 1 to 5 satisfaction score treats the gap between 1 and 2 as equal to the gap between 4 and 5, which nobody promised. Report the share of 4s and 5s or the median instead, or say clearly that you are treating the scale as interval.

#Skew, concentration and log scales

Right skew has a business meaning: a small share of rows carries a large share of the total. The quickest way to show it is "the top 1% of customers hold X% of revenue". When the data spans orders of magnitude, look at it on a log scale: a log turns multiplicative spread into additive spread, so a skewed variable often looks roughly symmetric after log10. The mean of the logs, transformed back, is the geometric mean, a sensible "typical value" for ratio data.

#Worked example: 7.6 million January taxi fares

The data is the New York City Taxi and Limousine Commission (TLC) yellow taxi trip records for January 2019, published by the TLC for public use. The TLC site serves recent months as Parquet; the DataTalksClub mirror on GitHub serves the 2019 files as CSV. Download it once (134 MB):

curl -LO https://github.com/DataTalksClub/nyc-tlc-data/releases/download/yellow/yellow_tripdata_2019-01.csv.gz
pip install duckdb pandas scipy

#Step 1: the summary table

import duckdb

trips = duckdb.sql("""
    select fare_amount, payment_type, trip_distance, PULocationID, DOLocationID,
           tpep_pickup_datetime, tpep_dropoff_datetime
    from 'yellow_tripdata_2019-01.csv.gz'
    where tpep_pickup_datetime >= '2019-01-01' and tpep_pickup_datetime < '2019-02-01'
""").df()
fares = trips["fare_amount"]

print("n =", f"{len(fares):,}")
print(fares.describe(percentiles=[.01, .25, .5, .75, .99]).round(2))
print("mode:", fares.mode().iloc[0])

n = 7,667,255
count    7667255.00
mean          12.41
std          262.08
min         -362.00
1%             3.00
25%            6.00
50%            8.50
75%           13.50
99%           52.00
max       623259.86
Name: fare_amount, dtype: float64
mode: 6.0

The where clause matters: the raw file has a handful of pickups dated 2001 and 2018 (device clock errors), so always filter to the period you claim to describe.

Read the table like an analyst:

• Mode 6.0 < median 8.50 < mean 12.41. Classic right skew: lots of short trips, a long tail of airport and out-of-town rides.

• The middle half of fares sits between 6.00 and 13.50 (IQR 7.50), yet the standard deviation is 262. A spread measure 35 times wider than the IQR means something extreme is hiding in the tail.

• The minimum is negative. Refunds and corrections are recorded as negative fares.

#Step 2: find the fare that breaks the average

print(trips.loc[fares.idxmax()].to_string())

without = fares.drop(fares.idxmax())
print("mean  with / without:", round(fares.mean(), 4), round(without.mean(), 4))
print("median with / without:", fares.median(), without.median())
print("sd    with / without:", round(fares.std(), 2), round(without.std(), 2))

fare_amount                        623259.86
payment_type                               3
trip_distance                            2.4
PULocationID                             237
DOLocationID                              90
tpep_pickup_datetime     2019-01-11 19:33:15
tpep_dropoff_datetime    2019-01-11 19:53:09
mean  with / without: 12.4092 12.328
median with / without: 8.5 8.5
sd    with / without: 262.08 134.26

A 2.4-mile, 20-minute ride on the Upper East Side, payment type 3 ("no charge"), with a fare of 623,259.86. It is a data entry error. One row out of 7.67 million moved the mean by 0.08 and halved the standard deviation. The median did not move at all.

The SD is still 134 without it, so there is more. Look at the top of the list before deciding anything:

cols = ["fare_amount", "payment_type", "trip_distance", "PULocationID", "DOLocationID"]
print(trips.nlargest(6, "fare_amount")[cols].to_string(index=False))
print("negative fares:", (fares < 0).sum(), " zero fares:", (fares == 0).sum())
print(trips.loc[fares < 0, "payment_type"].value_counts().sort_index().to_string())

 fare_amount  payment_type  trip_distance  PULocationID  DOLocationID
   623259.86             3            2.4           237            90
   355676.98             2            0.0            24           264
    36090.30             1            0.0           264           264
    34674.65             1            0.0           264           264
    33023.53             1            0.0           264           264
    31107.91             1            0.0           264           264
negative fares: 7131  zero fares: 2422
payment_type
1       1
2     376
3    4088
4    2666

Zone 264 is "Unknown" in the TLC zone lookup, and a distance of 0.0 means the meter recorded no movement. These are not trips; they are test records or errors. The negative fares cluster in payment types 3 (no charge) and 4 (dispute), which is what you would expect from voids and refunds. Now you have a reason to exclude them, written in business terms, not "they were outliers".

#Step 3: what the 1.5 IQR rule actually flags

q1, q3 = fares.quantile([0.25, 0.75])
iqr = q3 - q1
upper, lower = q3 + 1.5 * iqr, q1 - 1.5 * iqr
flagged = trips[(fares > upper) | (fares < lower)]
print(f"IQR = {iqr}, fences = [{lower}, {upper}]")
print(f"flagged: {len(flagged):,} trips ({len(flagged) / len(trips):.1%})")

airports = {132: "JFK", 138: "LaGuardia", 1: "Newark"}
touches_airport = flagged["PULocationID"].isin(airports) | flagged["DOLocationID"].isin(airports)
print(f"flagged trips that start or end at an airport: {touches_airport.mean():.1%}")

IQR = 7.5, fences = [-5.25, 24.75]
flagged: 706,505 trips (9.2%)
flagged trips that start or end at an airport: 60.9%

The rule flags 706,505 trips, and 61% of them touch an airport. A JFK run has a flat fare of 52 (that is the 99th percentile you saw above). These are the most valuable rides in the fleet. Deleting "outliers" by this rule would delete the airport business and make every revenue number wrong.

#Step 4: trimmed means, skew and concentration

import numpy as np
from scipy import stats

print("mean:", round(fares.mean(), 2),
      " 1% trimmed:", round(stats.trim_mean(fares, 0.01), 2),
      " 5% trimmed:", round(stats.trim_mean(fares, 0.05), 2),
      " median:", fares.median())

valid = fares[(fares > 0) & (fares < 500)]
print(f"valid fares kept: {len(valid):,} of {len(fares):,}")
print("skewness:", round(stats.skew(valid), 2))
top1 = valid.sort_values(ascending=False).head(len(valid) // 100)
print(f"share of all fare revenue from the top 1% of trips: {top1.sum() / valid.sum():.1%}")
logs = np.log10(valid)
print("log10 fare: mean", round(logs.mean(), 3), "median", round(logs.median(), 3),
      "skewness", round(stats.skew(logs), 2))
print("back-transformed mean of logs (geometric mean):", round(10 ** logs.mean(), 2))

mean: 12.41  1% trimmed: 11.78  5% trimmed: 10.61  median: 8.5
valid fares kept: 7,657,663 of 7,667,255
skewness: 4.25
share of all fare revenue from the top 1% of trips: 5.4%
log10 fare: mean 0.985 median 0.944 skewness 0.68
back-transformed mean of logs (geometric mean): 9.66

• The trimmed means walk from the mean towards the median as you trim more. That is the skew made visible.

• Skewness of 4.25 is strongly right-skewed (0 is symmetric; above 1 is usually called highly skewed). On the log scale it drops to 0.68 and the mean and median of the logs nearly agree.

• The top 1% of taxi trips earn only 5.4% of fare revenue, because metered fares have a natural ceiling. Compare customer spend in the UCI Online Retail data (CC BY 4.0), where the same calculation gives a very different answer:

import pandas as pd

url = "https://raw.githubusercontent.com/databricks/Spark-The-Definitive-Guide/master/data/retail-data/all/online-retail-dataset.csv"
retail = pd.read_csv(url, encoding="latin-1")
retail = retail[(retail["Quantity"] > 0) & (retail["UnitPrice"] > 0) & retail["CustomerID"].notna()]
spend = (retail["Quantity"] * retail["UnitPrice"]).groupby(retail["CustomerID"]).sum()
top = spend.sort_values(ascending=False).head(len(spend) // 100)
print(f"customers: {len(spend):,}")
print(f"top 1% of customers hold {top.sum() / spend.sum():.1%} of revenue")
print("mean spend", round(spend.mean(), 2), " median spend", round(spend.median(), 2))

customers: 4,338
top 1% of customers hold 31.8% of revenue
mean spend 2054.27  median spend 674.48

43 wholesale customers hold almost a third of revenue, and the mean customer spends three times the median customer. That is what "money data is skewed" looks like in a real business, and it is why "average customer value" is a dangerous KPI.

#Step 5: compare groups with the right summary

summary = duckdb.sql("""
    select payment_type,
           count(*)                                    as n,
           round(avg(fare_amount), 2)                  as mean,
           round(median(fare_amount), 2)               as median,
           quantile_cont(fare_amount, 0.25)            as p25,
           quantile_cont(fare_amount, 0.75)            as p75,
           round(stddev_samp(fare_amount), 2)          as sd
    from 'yellow_tripdata_2019-01.csv.gz'
    where tpep_pickup_datetime >= '2019-01-01' and tpep_pickup_datetime < '2019-02-01'
    group by payment_type
    order by payment_type
""").df()
summary["payment"] = summary["payment_type"].map(
    {1: "credit card", 2: "cash", 3: "no charge", 4: "dispute", 5: "unknown"})
print(summary.to_string(index=False))

 payment_type       n  mean  median  p25  p75      sd     payment
            1 5485742 12.70     9.0 6.50 14.0   43.73 credit card
            2 2137163 11.43     8.0 5.50 12.5  243.56        cash
            3   33186 29.70     6.0 2.50 12.5 3425.29   no charge
            4   11164  7.74     6.0 0.01 12.0   21.39     dispute

Look at "no charge": a mean of 29.70 against a median of 6.00. Someone skimming means would conclude no-charge trips are the most expensive. The median says they are the cheapest, and the mean is the 623,259.86 row talking. Card trips are a little longer than cash trips on every percentile, which is a real and stable difference.

So what goes in the report? Something like:

In January 2019 the typical yellow taxi fare was 8.50 (median; middle half between 6.00 and 13.50; n = 7,657,663 trips). We excluded 9,592 records with zero, negative or impossible fares (above 500), mostly refunds, disputes and meter tests.

#Common mistakes

• Quoting the mean of skewed data alone. Income, order value, delivery time, session length: quote the median, or both with the reason they differ.

• Deleting by a rule instead of by a reason. The 1.5 IQR rule would have removed every airport trip. Investigate the flagged rows, then exclude only what you can name: test records, refunds, impossible values.

• Summaries without n. "Median 8.50" could be from 12 trips or 7 million. Always print the count, and the count you excluded.

• Averaging codes and ratings. mean(payment_type) is meaningless; the mean of a 1 to 5 rating is an assumption. Check the data type first.

• Using the SD to describe skewed spread. An SD of 262 on fares whose middle half spans 7.50 describes the errors, not the trips. Use percentiles.

• Forgetting the date filter. Raw logs contain rows from the wrong period. Filter to the period your sentence claims.

#Interview angle

#Practice task

Use the same January 2019 taxi file.

1. Produce the Step 1 summary table for trip_distance. What is the maximum, and is it plausible for a city taxi?

2. Find every trip with trip_distance above 100 miles. Write one sentence on what they are, based on the rows.

3. Compare median and IQR of fares for pickups at JFK (PULocationID = 132) and Midtown Center (161). Report n for each.

4. Compute the share of trips flagged by the 1.5 IQR rule for tip_amount among card payments, and say whether you would remove them.

5. Write the two-sentence summary you would send a manager about January trip distances, including exclusions.

Done when you have a table, two sentences and an exclusion count you can defend.

dist = trips["trip_distance"]
print(dist.describe(percentiles=[.25, .5, .75, .99]).round(2))
print(trips[dist > 100].sort_values("trip_distance", ascending=False).head(10))

jfk = trips.loc[trips["PULocationID"] == 132, "fare_amount"]
mid = trips.loc[trips["PULocationID"] == 161, "fare_amount"]
for name, s in [("JFK", jfk), ("Midtown Center", mid)]:
    q1, med, q3 = s.quantile([.25, .5, .75])
    print(name, "n =", len(s), "median =", med, "IQR =", q3 - q1)

#Further reading

• NIST/SEMATECH e-Handbook: measures of location: definitions of mean, median, mode and trimmed mean, with when each is robust.

• NYC TLC trip record data: the source files and the data dictionary that defines payment types and rate codes.

• SciPy trim_mean: exact trimming behaviour.

• Stevens, On the Theory of Scales of Measurement (Science, 1946): the original nominal, ordinal, interval and ratio paper.