Python · Beginner

Python dictionaries and sets with real examples

14 min read

Why it matters

This Python dictionary tutorial is about the data structure you will reach for more than any other. Matching records, counting things, grouping rows, caching results and reading JSON from an API are all dictionary work, and comparing two lists of IDs ("which customers did we lose?") is set work. Interviewers know it: "two-sum", "group anagrams" and "first non-repeating character" are all dictionary questions in disguise, and they ask them in analyst, data engineering and software screening rounds alike.

You will compare two years of a real online retailer's customers, rank its products, and learn the rules that make dictionaries fast and the ones that make them surprising.

The concept

Keys must be hashable

A dictionary stores each key in a slot chosen by the key's hash, a number computed from its value. That is why lookup is O(1) on average: Python computes the hash and jumps to the slot instead of scanning. It also explains the rule: a key's hash must never change, so keys must be immutable (strings, numbers, tuples of those). Lists and dicts cannot be keys:

dict_basics.py

stock = {"JUMBO BAG RED RETROSPOT": 120, "PARTY BUNTING": 40}

print(stock.get("POPCORN HOLDER"), stock.get("POPCORN HOLDER", 0))
try:
    stock["POPCORN HOLDER"]
except KeyError as e:
    print("KeyError:", e)

orders = {}
orders.setdefault("13085", []).append("489434")    # creates the list the first time
orders.setdefault("13085", []).append("489435")    # reuses it afterwards
print(orders)

key = ("United Kingdom", "13085")                  # a tuple of strings is hashable
visits = {key: 2}
try:
    visits[["United Kingdom", "13085"]] = 3
except TypeError as e:
    print("TypeError:", e)

print(list({"b": 1, "a": 2, "c": 3}))              # insertion order is kept (3.7+)

for name, qty in stock.items():
    print(f"{name:<25}{qty:>4}")

try:
    for name in stock:
        if stock[name] < 50:
            del stock[name]
except RuntimeError as e:
    print("RuntimeError:", e)
stock = {name: qty for name, qty in stock.items() if qty >= 50}
print(stock)

text

None 0
KeyError: 'POPCORN HOLDER'
{'13085': ['489434', '489435']}
TypeError: unhashable type: 'list'
['b', 'a', 'c']
JUMBO BAG RED RETROSPOT   120
PARTY BUNTING              40
RuntimeError: dictionary changed size during iteration
{'JUMBO BAG RED RETROSPOT': 120}

Reading values: [], get and setdefault

expression	key missing	use it when
`d[key]`	raises `KeyError`	the key must exist; a missing key is a bug you want to see
`d.get(key)`	returns `None`	absence is normal
`d.get(key, default)`	returns `default`	you have a sensible fallback, such as `0`
`d.setdefault(key, [])`	inserts the default and returns it	building groups in a plain dict
`defaultdict(list)`	creates the default automatically	building groups, cleaner than `setdefault`

Do not reach for get everywhere to "avoid errors". If a config key must exist, d["key"] failing loudly is better than None flowing through your program.

Order, iteration and changing size

Since Python 3.7, dictionaries keep insertion order. Iterating gives keys in the order they were first added. (Sets do not keep any order.)
for k in d loops over keys, d.values() over values, d.items() over (key, value) pairs. Use items() when you need both; it avoids a second lookup.
Adding or deleting keys while iterating raises RuntimeError: dictionary changed size during iteration. Build a new dict with a comprehension, or loop over list(d).

Counter and defaultdict

collections.Counter is a dictionary specialised for counting. Counter(iterable) counts in one go, c[key] += n works without initialising, missing keys read as 0, and most_common(n) returns the top n. Counters also support arithmetic:

python

from collections import Counter

a = Counter(red=3, blue=1)
b = Counter(red=1, green=2)
print(a + b, a - b, a & b, a | b)

text

Counter({'red': 4, 'green': 2, 'blue': 1}) Counter({'red': 2, 'blue': 1}) Counter({'red': 1}) Counter({'red': 3, 'green': 2, 'blue': 1})

- subtracts and drops anything that ends at zero or below; & keeps the minimum; | the maximum. That makes "sold minus returned" a single expression, as you will see on real data below.

defaultdict(list) and defaultdict(int) (from Python basics for beginners on a real dataset) handle grouping and simple counts.

Merging dictionaries

merge_invert.py

defaults = {"currency": "GBP", "country": "United Kingdom", "vat": 0.2}
india = {"currency": "INR", "country": "India", "vat": None}

print(defaults | india)            # right side wins on clashes (3.9+)
print({**india, **defaults})       # same idea, older syntax: the last one wins
settings = dict(defaults)
settings |= {"vat": 0.18}          # in-place update, like settings.update(...)
print(settings)

country_of = {"12346": "United Kingdom", "12347": "Iceland", "12348": "Finland", "12349": "Italy",
              "12350": "Norway", "12352": "Norway"}
squares = {n: n * n for n in range(4)}
print(squares)
print({v: k for k, v in country_of.items()})       # duplicates: last key silently wins

customers_by_country = {}
for customer, country in country_of.items():
    customers_by_country.setdefault(country, []).append(customer)
print(customers_by_country)
print(sorted({c for c in country_of.values()}))     # a set comprehension removes duplicates

text

{'currency': 'INR', 'country': 'India', 'vat': None}
{'currency': 'GBP', 'country': 'United Kingdom', 'vat': 0.2}
{'currency': 'GBP', 'country': 'United Kingdom', 'vat': 0.18}
{0: 0, 1: 1, 2: 4, 3: 9}
{'United Kingdom': '12346', 'Iceland': '12347', 'Finland': '12348', 'Italy': '12349', 'Norway': '12352'}
{'United Kingdom': ['12346'], 'Iceland': ['12347'], 'Finland': ['12348'], 'Italy': ['12349'], 'Norway': ['12350', '12352']}
['Finland', 'Iceland', 'Italy', 'Norway', 'United Kingdom']

In every merge form, the right-hand (later) value wins. Note that None wins too: india's "vat": None replaced 0.2. If None should mean "not set", filter it out before merging.
update() and |= change the dict in place; | and {**a, **b} build a new one.
Inverting a dictionary with a comprehension loses data when values repeat: two Norwegian customers became one. Invert into lists (setdefault(v, []).append(k)) when values are not unique. (Those six customer IDs and countries are the first six in the retail data used below.)

Sets

A set is a collection of unique, hashable items with no order and O(1) average membership tests. Its operators are exactly the questions you ask when comparing two lists of IDs:

operator	method	meaning	question it answers
`a & b`	`a.intersection(b)`	in both	returning customers
`a - b`	`a.difference(b)`	in a, not b	lost customers
`b - a`		in b, not a	new customers
`a \| b`	`a.union(b)`	in either	everyone we have ever served
`a ^ b`	`a.symmetric_difference(b)`	in exactly one	everyone whose status changed

Use set() for an empty set; {} is an empty dict. Printing a set of strings can show a different order on every run (string hashes are randomised per process), so sort before you print or compare output.

Why O(1) lookup matters: matching records

Matching each invoice line to its customer with nested loops compares every line with every customer. Building a dictionary once turns each match into a single lookup. On 20,000 real invoice lines and 756 customers:

matching.py

import csv
import time

with open("../retail/online_retail_2010_2011.csv", newline="", encoding="utf-8") as f:
    lines = [r for r in csv.DictReader(f) if r["Customer ID"]][:20_000]

# A "customers" table: one row per customer ID with its country.
customers = [{"id": cid, "country": None} for cid in sorted({r["Customer ID"] for r in lines})]
for c in customers:
    c["country"] = next(r["Country"] for r in lines if r["Customer ID"] == c["id"])
print(len(lines), "invoice lines,", len(customers), "customers")

start = time.perf_counter()
matched = 0
for line in lines:                                  # nested loops: O(lines x customers)
    for c in customers:
        if c["id"] == line["Customer ID"]:
            matched += 1
            break
print(f"nested loops : {matched} matched in {time.perf_counter() - start:.3f} s")

start = time.perf_counter()
country_of = {c["id"]: c["country"] for c in customers}   # build once: O(customers)
matched = sum(1 for line in lines if line["Customer ID"] in country_of)   # O(lines)
print(f"dict lookup  : {matched} matched in {time.perf_counter() - start:.4f} s")

text

20000 invoice lines, 756 customers
nested loops : 20000 matched in 0.612 s
dict lookup  : 20000 matched in 0.0029 s

About 200 times faster, and the gap grows with the data: double both inputs and the nested version takes four times as long, the dictionary version twice as long.

Nested dictionaries from JSON

APIs return nested JSON, which json.loads turns into nested dicts and lists. Two different things can go wrong at each level: the key is missing (KeyError), or it is present with the value null, which becomes None (TypeError when you index into it).

nested_json.py

import json

# Part of the real response from https://pypi.org/pypi/requests/2.32.3/json
text = """{"info": {"name": "requests", "version": "2.32.3", "requires_python": ">=3.8",
          "maintainer": null,
          "project_urls": {"Documentation": "https://requests.readthedocs.io",
                           "Source": "https://github.com/psf/requests"}}}"""
data = json.loads(text)

print(data["info"]["project_urls"]["Source"])
print(data["info"].get("project_urls", {}).get("Changelog", "no changelog link"))

try:
    data["info"]["stars"]
except KeyError as e:
    print("KeyError:", e)                 # the key is missing
try:
    data["info"]["maintainer"]["email"]
except TypeError as e:
    print("TypeError:", e)                # the key exists but holds null (None)


def dig(obj, *keys, default=None):
    """Follow keys through nested dicts; return default at the first missing or None level."""
    for key in keys:
        if not isinstance(obj, dict) or obj.get(key) is None:
            return default
        obj = obj[key]
    return obj

print(dig(data, "info", "project_urls", "Source"))
print(dig(data, "info", "maintainer", "email", default="unknown"))

text

https://github.com/psf/requests
no changelog link
KeyError: 'stars'
TypeError: 'NoneType' object is not subscriptable
https://github.com/psf/requests
unknown

Chained .get(key, {}) works when keys are missing, but not when a value is null: None.get fails. A small helper like dig handles both, and for anything larger, validate the whole shape with a model (Pydantic is covered in the dataclasses lesson).

Worked example: new, lost and returning customers

The UCI Online Retail II dataset (CC BY 4.0, UCI repository) holds every transaction of a UK online gift retailer from December 2009 to December 2011, published as an Excel workbook with one sheet per year. Save each sheet as CSV (online_retail_2009_2010.csv with 525,461 rows and online_retail_2010_2011.csv with 541,910 rows). The columns are Invoice, StockCode, Description, Quantity, InvoiceDate, Price, Customer ID, Country.

Step 1: check how the two files relate

Before comparing "year 1" with "year 2", check whether the files really are separate years. Both contain invoices from 1 to 9 December 2010:

overlap.py

import csv
from collections import Counter


def rows_between(path, start, end):
    with open(path, newline="", encoding="utf-8") as f:
        return [tuple(r.values()) for r in csv.DictReader(f) if start <= r["InvoiceDate"] < end]


a = rows_between("online_retail_2009_2010.csv", "2010-12-01", "2010-12-10")
b = rows_between("online_retail_2010_2011.csv", "2010-12-01", "2010-12-10")
print(len(a), len(b), Counter(a) == Counter(b))

text

22523 22523 True

The same 22,523 rows appear in both sheets (in a different order, which is why the rows are compared as Counters of tuples, a multiset comparison). Comparing the files as they are would count every customer from those nine days as "returning". So define the years by date: year 1 is December 2009 to November 2010, year 2 is December 2010 onwards. Finding this kind of overlap before you publish a number is what separates an analyst people trust from one they double-check.

Step 2: compare the customer sets

customers.py

import csv
from collections import Counter


def customer_ids(path, start, end):
    """Customer IDs with an invoice in [start, end). Blank IDs (guest checkouts) are skipped."""
    ids = set()
    with open(path, newline="", encoding="utf-8") as f:
        for row in csv.DictReader(f):
            raw = row["Customer ID"]
            if raw and start <= row["InvoiceDate"] < end:   # ISO dates compare correctly as text
                ids.add(raw.removesuffix(".0"))            # the export writes 13085 as "13085.0"
    return ids


# The two sheets overlap on 1 to 9 December 2010, so define the years by date, not by file.
year1 = customer_ids("online_retail_2009_2010.csv", "2009-12-01", "2010-12-01")
year2 = customer_ids("online_retail_2010_2011.csv", "2010-12-01", "2011-12-10")

returning = year1 & year2          # in both years
lost = year1 - year2               # bought in year 1, not in year 2
new = year2 - year1                # first seen in year 2
either = year1 | year2
changed = year1 ^ year2            # in exactly one year (lost plus new)

print(f"year 1 customers : {len(year1):>5}")
print(f"year 2 customers : {len(year2):>5}")
print(f"returning        : {len(returning):>5}  ({len(returning) / len(year1):.1%} of year 1)")
print(f"lost             : {len(lost):>5}")
print(f"new              : {len(new):>5}")
print(f"either year      : {len(either):>5}")
print(f"exactly one year : {len(changed):>5} = lost + new: {len(changed) == len(lost) + len(new)}")

text

year 1 customers :  4336
year 2 customers :  4372
returning        :  2766  (63.8% of year 1)
lost             :  1570
new              :  1606
either year      :  5942
exactly one year :  3176 = lost + new: True

Reading it as a business answer: about 64% of year-one customers came back, 1,570 did not, and 1,606 new customers more than replaced them. The last line is a self-check: two ways of computing the same thing must agree. Build checks like that into every analysis.

Two details in customer_ids matter. The file stores IDs as "13085.0" because the column was numeric in Excel; if one file had "13085" and the other "13085.0", every set operation would silently say "no overlap". Normalise IDs before comparing. And rows with a blank ID are guest checkouts that cannot be tracked, so they are skipped on purpose.

Step 3: top products with Counter

products.py

import csv
from collections import Counter

sold = Counter()        # product -> units on normal invoices
cancelled = Counter()   # product -> units on cancellation invoices (Invoice starts with "C")
with open("online_retail_2010_2011.csv", newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        name = row["Description"].strip()
        qty = int(row["Quantity"])
        if row["Invoice"].startswith("C"):
            cancelled[name] += -qty          # cancellation quantities are negative in the file
        else:
            sold[name] += qty

print("Top 3 by units sold (ignoring cancellations):")
for name, n in sold.most_common(3):
    print(f"  {n:>6}  {name}")

net = sold - cancelled      # Counter subtraction drops anything that ends at zero or below
print("Top 3 by net units:")
for name, n in net.most_common(3):
    print(f"  {n:>6}  {name}")
print("PAPER CRAFT , LITTLE BIRDIE:", sold["PAPER CRAFT , LITTLE BIRDIE"], "sold,",
      cancelled["PAPER CRAFT , LITTLE BIRDIE"], "cancelled, net", net["PAPER CRAFT , LITTLE BIRDIE"])

text

Top 3 by units sold (ignoring cancellations):
   80995  PAPER CRAFT , LITTLE BIRDIE
   78033  MEDIUM CERAMIC TOP STORAGE JAR
   55047  WORLD WAR 2 GLIDERS ASSTD DESIGNS
Top 3 by net units:
   53847  WORLD WAR 2 GLIDERS ASSTD DESIGNS
   47363  JUMBO BAG RED RETROSPOT
   36381  ASSORTED COLOUR BIRD ORNAMENT
PAPER CRAFT , LITTLE BIRDIE: 80995 sold, 80995 cancelled, net 0

The naive "top product" is a single 80,995-unit order that was cancelled in full. sold - cancelled removes it in one expression, and the real best-sellers appear. Whenever one item dominates a ranking, look at the rows behind it before you report it.

Common mistakes

Using d.get() everywhere. It hides missing data that should have stopped the program. Use d[key] for keys that must exist.
Deleting keys while looping over the dict. RuntimeError. Build a new dict with a comprehension.
Expecting {} to be an empty set. It is an empty dict; use set().
Inverting a dict with non-unique values. The comprehension keeps only the last key per value. Invert into lists.
Merging and forgetting None wins. A later None overwrites a real value. Drop Nones first if they mean "unset".
Comparing IDs with different formats. "13085" and "13085.0" (or 13085 the integer) never match. Normalise to one string format.
Comparing overlapping extracts. Check date ranges before computing new versus returning customers.
Printing sets in tests or reports. Order changes between runs. Sort first.
Nested loops to match records. O(n×m). Build a dict or set once.

Interview angle

Find the first non-repeating character, group anagrams, and two-sum.

interview.py

from collections import Counter, defaultdict


def first_unique(text):
    counts = Counter(text)
    for ch in text:
        if counts[ch] == 1:
            return ch
    return None


def group_anagrams(words):
    groups = defaultdict(list)
    for w in words:
        groups["".join(sorted(w))].append(w)
    return list(groups.values())


def two_sum(nums, target):
    seen = {}                       # value -> index
    for i, n in enumerate(nums):
        if target - n in seen:
            return seen[target - n], i
        seen[n] = i
    return None


print(first_unique("swiss"), group_anagrams(["listen", "silent", "enlist", "google", "gogole", "cat"]),
      two_sum([2, 7, 11, 15], 9))

text

w [['listen', 'silent', 'enlist'], ['google', 'gogole'], ['cat']] (0, 1)

The explanation is the same each time: one pass, a dictionary remembers what you have seen, each lookup is O(1) on average, so the whole thing is O(n) instead of the O(n²) of comparing every pair. For anagrams, the sorted letters are the key (O(k log k) per word).

Why can a tuple be a dictionary key but a list cannot?

Keys are stored by their hash, and the hash must not change while the key is in the dict. Tuples (of hashable items) are immutable, so their hash is stable. Lists can change, so they are unhashable. A tuple containing a list is also unhashable.

Is dictionary lookup always O(1)?

On average, yes. In the worst case (many keys with colliding hashes) it degrades towards O(n), which is why Python randomises string hashes per process: it stops attackers from crafting colliding keys. In practice you treat it as O(1).

Practice task

Use the two Online Retail II CSV files.

Find the countries that had customers in year 1 but none in year 2, and the countries that appear only in year 2. Use sets of country names with the same date windows as above.
For the 1,570 lost customers, count which country they came from with Counter and print the top 5.
Build invoices_by_customer: a dict mapping each year-2 customer ID to the set of their invoice numbers (exclude cancellations). Print the 5 customers with the most invoices.
Using country_of from the matching example as a pattern, write a function that takes a list of customer IDs and returns a dict {country: [ids...]}, using setdefault or defaultdict(list).

Done when: every answer prints, your step 2 counts add up to 1,570, and you can explain why you used a set or a dict at each step.

Show hints

Step 1 is countries1 - countries2 and countries2 - countries1.
Step 2: build country_of for year 1 first (customer ID to country), then Counter(country_of[c] for c in lost). Check sum(counter.values()) == len(lost).
Step 3: defaultdict(set), then sorted(d.items(), key=lambda kv: len(kv[1]), reverse=True)[:5].
A few customers have invoices from more than one country; decide whether you keep the first or the most frequent, and say which in a comment.

#Why it matters

#The concept

#Keys must be hashable

#Reading values: [], get and setdefault

#Order, iteration and changing size

#Counter and defaultdict

#Merging dictionaries

#Sets

#Why O(1) lookup matters: matching records

#Nested dictionaries from JSON

#Worked example: new, lost and returning customers

#Step 1: check how the two files relate

#Step 2: compare the customer sets

#Step 3: top products with Counter

#Common mistakes

#Interview angle

#Practice task

#Further reading

Why it matters

The concept

Keys must be hashable

Reading values: [], get and setdefault

Order, iteration and changing size

Counter and defaultdict

Merging dictionaries

Sets

Why O(1) lookup matters: matching records

Nested dictionaries from JSON

Worked example: new, lost and returning customers

Step 1: check how the two files relate

Step 2: compare the customer sets

Step 3: top products with Counter

Common mistakes

Interview angle

Practice task

Further reading