Y

Python · Intermediate

Call an API from a script you can hand over

3 min read

Why it matters

"Automate data collection from APIs" and "build scripts to integrate data sources" show up in analyst, data engineering and forward deployed engineering job descriptions. A script that works only on your laptop is not automation. It needs its own environment, clear errors and output someone else can use.

The concept

  1. Virtual environments isolate each project's packages, so upgrading pandas for one project does not break another. Create one per project and record the packages in requirements.txt.

  2. HTTP APIs return data, usually JSON, from a URL. You send a request, check the status code, then parse the body.

  3. Errors will happen: timeouts, rate limits, bad responses. Decide which to retry, which to report and which to stop on.

  4. Scripts take arguments instead of hard-coded values, so they can be reused.

Worked example: GDP per capita from the World Bank

The World Bank publishes its indicators under CC BY 4.0 through a free API with no key.

setup.sh
python3 -m venv .venv
source .venv/bin/activate          # on Windows: .venv\Scripts\activate
pip install requests
pip freeze > requirements.txt
gdp.py
"""Fetch GDP per capita (current USD) for a few countries from the World Bank API."""
import argparse
import csv
import sys
import time

import requests

API = "https://api.worldbank.org/v2/country/{countries}/indicator/NY.GDP.PCAP.CD"


def fetch(countries: list[str], years: str, retries: int = 3) -> list[dict]:
    url = API.format(countries=";".join(countries))
    params = {"format": "json", "date": years, "per_page": 500}
    for attempt in range(1, retries + 1):
        try:
            response = requests.get(url, params=params, timeout=20)
            response.raise_for_status()
            meta, rows = response.json()
            return [
                {"country": r["country"]["value"], "year": int(r["date"]), "gdp_per_capita": r["value"]}
                for r in rows
                if r["value"] is not None
            ]
        except (requests.Timeout, requests.ConnectionError) as error:
            print(f"attempt {attempt} failed: {error}", file=sys.stderr)
            time.sleep(2 ** attempt)
    sys.exit("World Bank API unreachable, giving up")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("countries", nargs="+", help="ISO3 codes, e.g. IND CHN USA")
    parser.add_argument("--years", default="2020:2024")
    parser.add_argument("--out", default="gdp.csv")
    args = parser.parse_args()

    rows = fetch(args.countries, args.years)
    with open(args.out, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=["country", "year", "gdp_per_capita"])
        writer.writeheader()
        writer.writerows(sorted(rows, key=lambda r: (r["country"], r["year"])))
    print(f"wrote {len(rows)} rows to {args.out}")
run.sh
python gdp.py IND CHN USA --years 2020:2024

The API's data was last updated on 13 July 2026 when we ran this. India's GDP per capita rises from about 1,907 USD in 2020 to about 2,592 USD in 2024; China's from about 10,627 to 13,293.

Notice what the script does for its next user: --help explains it, missing values are skipped on purpose, network errors are retried with a growing wait, and a permanent failure exits with a clear message instead of a stack trace.

Common mistakes

  • No timeout. requests.get(url) can hang forever. Always pass timeout.

  • Retrying everything. Retry timeouts and 5xx errors. Do not retry a 400 or 404; the request itself is wrong.

  • Secrets in code. API keys go in environment variables (os.environ["API_KEY"]), never in the script or the repository.

  • Installing packages globally. Sooner or later two projects need different versions. Use a virtual environment every time.

Interview angle

"How would you pull data from an API every day?" Mention: virtual environment and pinned requirements, timeouts and retries with backoff, handling pagination, storing raw responses before transforming, and a scheduler (cron, Airflow or GitHub Actions). That covers what the interviewer is listening for.

Practice task

  1. Add a --indicator argument so the same script can fetch life expectancy (SP.DYN.LE00.IN).

  2. Handle pagination: request per_page=50 and loop until you have all pages (meta["pages"]).

  3. Write the raw JSON to a raw/ folder before transforming, named by date.

Further reading